Applying diagnostic thresholds for request-level metrics

[FabianMeiswinkel]

Description

This PR adds the option to filter request-level metrics via Diagnostics Thresholds. Request-level metrics use a dimension for the Service Endpoint / and/or Replica - s0, they allow to help debugging any transport/backend related issues - because they can help identify whether connectivity issues and/or high latency are constrained to a certain backend endpoint or not. For workloads with large containers (using hundreds of physical partitions) this means very high cardinality dimensions on those metrics - which can put a lot of pressure on the monitoring system due the high number of timeseries stored for these metrics. The filtering by diagnostics thresholds will reduce the overhead here significantly, because request-level metrics will be sampled and only be emitted when operations violate the expected thresholds (latency, RU charge etc.) - so, they are available to debug any threshold violations - but the cardinality will be lower at least when these threshold violations don't happen across all service endpoints / backend replica simultaneously - in which case that is a clear sign that the problem is on the client-side.

How to enable diagnostic threshold sampling for request-level metrics?

MeterRegistry meterRegistry = ConsoleLoggingRegistryFactory.create(1);

CosmosClientBuilder builder = this
    .getClientBuilder()
    .clientTelemetryConfig(
        new CosmosClientTelemetryConfig()
            .diagnosticsHandler(CosmosDiagnosticsHandler.DEFAULT_LOGGING_HANDLER)
            .diagnosticsThresholds(
                new CosmosDiagnosticsThresholds()
                    .setPointOperationLatencyThreshold(Duration.ofSeconds(3))
            )
            .metricsOptions(new CosmosMicrometerMetricsOptions()
                .meterRegistry(meterRegistry)
                .applyDiagnosticThresholdsForTransportLevelMeters(true))
    );

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[azure-sdk]

API change check

APIView has identified API level changes in this PR and created following API reviews.

azure-cosmos

[xinlian12]

LGTM, thanks