Support of high resolution metrics for CloudWatch

[mpelnens]

At the moment, the CloudWatch implementation publishes MetricDatum with the default StorageResolution of 1 minute. It would be great to be able to configure the resolution. There are use cases where 1 minute is too coarse, for example, when monitoring internal queues. CloudWatch supports storage resolutions of down to 1 second.

[jkschneider]

@COGNITIFF I believe it is already possible by configuring CloudwatchConfig#step() which simply defaults to 1 minute.

[mpelnens]

I assumed so too, but "step" only affects the push frequency. Even if data is pushed to AWS at a higher rate, CloudWatch, by default, still rolls it up to 1-minute increments. Enabling more granular storage and analysis requires additionally setting StorageResolution to a lower value, which activates what AWS calls "high-resolution metrics" (comes at a cost, by the way).

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html#high-resolution-metrics

[jkschneider]

@COGNITIFF Thanks for the explanation. I see now that there is a storageResolution method on MetricDatum.Builder that we can set to either 1 or 60:

@Nullable
private MetricDatum metricDatum(Meter.Id id, @Nullable String suffix, StandardUnit standardUnit, double value) {
    if (Double.isNaN(value)) {
        return null;
    }

    List<Tag> tags = id.getConventionTags(config().namingConvention());
    return MetricDatum.builder()
            .storageResolution(1) // or 60
            .metricName(getMetricName(id, suffix))
            .dimensions(toDimensions(tags))
            .timestamp(timestamp)
            .value(CloudWatchUtils.clampMetricValue(value))
            .unit(standardUnit)
            .build();
}

So this is tricky, being a per-metric decision. Do you want all of your metrics high resolution or just some subset?

[mpelnens]

I think a registry-scoped setting would suffice, especially since "step" is also registry scoped, and they are interrelated.

[jkschneider]

@COGNITIFF Do you suppose that if step is less than 1 minute, then we should always use high-resolution?

[mpelnens]

It would reduce configuration effort, but I can think of use-cases where users (especially for cost-saving reasons) would be ok with more granular data being rolled-up and reported by CloudWatch at 1-minute intervals. Keeping the two settings separate would allow for more flexibility.

[jkschneider]

@COGNITIFF Thanks. We do have a config complexity/flexibility tradeoff to evaluate. I was thinking that maybe we stick with your original suggestion to do this on a per-registry basis for now. Added a note about how a user could only send some metrics at high resolution by using deny filters and two CloudwatchMeterRegistry. WDYT?

This config option allows you to override the "if step <1 minute do high resolution" thing, btw.

[mpelnens]

That's a great solution, and I think it will address most scenarios! Thanks!