Skip to content

Commit

Permalink
[sdk-metrics] Obsolete SetMaxMetricPointsPerMetricStream + standarize…
Browse files Browse the repository at this point in the history 
… on "Cardinality Limit" name (#5328)

Co-authored-by: Yun-Ting Lin <yunl@microsoft.com>
  • Loading branch information
@CodeBlanch @Yun-Ting
CodeBlanch and Yun-Ting committed Feb 9, 2024
1 parent f214d27 commit cf00e42
Show file tree
Hide file tree
Showing 16 changed files with 190 additions and 232 deletions.
38 changes: 28 additions & 10 deletions docs/diagnostics/experimental-apis/OTEL1003.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,37 @@ Experimental APIs may be changed or removed in the future.

## Details

The OpenTelemetry Specification defines the
[cardinality limit](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#cardinality-limits)
of a metric can be set by the matching view.

From the specification:

> The cardinality limit for an aggregation is defined in one of three ways:
> A view with criteria matching the instrument an aggregation is created for has
> an aggregation_cardinality_limit value defined for the stream, that value
> SHOULD be used. If there is no matching view, but the MetricReader defines a
> default cardinality limit value based on the instrument an aggregation is
> created for, that value SHOULD be used. If none of the previous values are
> defined, the default value of 2000 SHOULD be used.
>
> 1. A view with criteria matching the instrument an aggregation is created for
> has an `aggregation_cardinality_limit` value defined for the stream, that
> value SHOULD be used.
> 2. If there is no matching view, but the `MetricReader` defines a default
> cardinality limit value based on the instrument an aggregation is created
> for, that value SHOULD be used.
> 3. If none of the previous values are defined, the default value of 2000
> SHOULD be used.
We are exposing these APIs experimentally until the specification declares them
stable.

### Setting cardinality limit for a specific Metric via the View API

The OpenTelemetry Specification defines the [cardinality
limit](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#cardinality-limits)
of a metric can be set by the matching view.

```csharp
using var meterProvider = Sdk.CreateMeterProviderBuilder()
.AddView(
instrumentName: "MyFruitCounter",
new MetricStreamConfiguration { CardinalityLimit = 10 })
.Build();
```

### Setting cardinality limit for a specific MetricReader

[This is not currently supported by OpenTelemetry
.NET.](https://github.com/open-telemetry/opentelemetry-dotnet/issues/5331)
18 changes: 10 additions & 8 deletions docs/metrics/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -379,17 +379,19 @@ predictable and reliable behavior when excessive cardinality happens, whether it
was due to a malicious attack or developer making mistakes while writing code.

OpenTelemetry has a default cardinality limit of `2000` per metric. This limit
can be configured at `MeterProvider` level using the
`SetMaxMetricPointsPerMetricStream` method, or at individual
[view](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#view)
level using `MetricStreamConfiguration.CardinalityLimit`. Refer to this
[doc](../../docs/metrics/customizing-the-sdk/README.md#changing-maximum-metricpoints-per-metricstream)
can be configured at the individual metric level using the [View
API](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#view)
and the `MetricStreamConfiguration.CardinalityLimit` setting. Refer to this
[doc](../../docs/metrics/customizing-the-sdk/README.md#changing-the-cardinality-limit-for-a-metric)
for more information.

Given a metric, once the cardinality limit is reached, any new measurement which
cannot be independently aggregated because of the limit will be aggregated using
the [overflow
attribute](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#overflow-attribute).
cannot be independently aggregated because of the limit will be dropped or
aggregated using the [overflow
attribute](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#overflow-attribute)
(if enabled). When NOT using the overflow attribute feature a warning is written
to the [self-diagnostic log](../../src/OpenTelemetry/README.md#self-diagnostics)
the first time an overflow is detected for a given metric.

> [!NOTE]
> Overflow attribute was introduced in OpenTelemetry .NET
Expand Down
88 changes: 11 additions & 77 deletions docs/metrics/customizing-the-sdk/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -367,90 +367,24 @@ MyFruitCounter.Add(1, new("name", "apple"), new("color", "red"));
AnotherFruitCounter.Add(1, new("name", "apple"), new("color", "red"));
```

### Changing maximum MetricPoints per MetricStream
### Changing the cardinality limit for a Metric

A Metric stream can contain as many Metric points as the number of unique
combination of keys and values. To protect the SDK from unbounded memory usage,
SDK limits the maximum number of metric points per metric stream, to a default
of 2000. Once the limit is hit, any new key/value combination for that metric is
ignored. The SDK chooses the key/value combinations in the order in which they
are emitted. `SetMaxMetricPointsPerMetricStream` can be used to override the
default.
To set the [cardinality limit](../README.md#cardinality-limits) for an
individual metric, use `MetricStreamConfiguration.CardinalityLimit` setting on
the View API:

> [!NOTE]
> One `MetricPoint` is reserved for every `MetricStream` for the
special case where there is no key/value pair associated with the metric. The
maximum number of `MetricPoint`s has to accommodate for this special case.

Consider the below example. Here we set the maximum number of `MetricPoint`s
allowed to be `3`. This means that for every `MetricStream`, the SDK will export
measurements for up to `3` distinct key/value combinations of the metric. There
are two instruments published here: `MyFruitCounter` and `AnotherFruitCounter`.
There are two total `MetricStream`s created one for each of these instruments.
SDK will limit the maximum number of distinct key/value combinations for each of
these `MetricStream`s to `3`.

```csharp
using System.Collections.Generic;
using System.Diagnostics.Metrics;
using OpenTelemetry;
using OpenTelemetry.Metrics;

Counter<long> MyFruitCounter = MyMeter.CreateCounter<long>("MyFruitCounter");
Counter<long> AnotherFruitCounter = MyMeter.CreateCounter<long>("AnotherFruitCounter");

using var meterProvider = Sdk.CreateMeterProviderBuilder()
.AddMeter("*")
.AddConsoleExporter()
.SetMaxMetricPointsPerMetricStream(3) // The default value is 2000
.Build();

// There are four distinct key/value combinations emitted for `MyFruitCounter`:
// 1. No key/value pair
// 2. (name:apple, color:red)
// 3. (name:lemon, color:yellow)
// 4. (name:apple, color:green)
// Since the maximum number of `MetricPoint`s allowed is `3`, the SDK will only export measurements for the following three combinations:
// 1. No key/value pair
// 2. (name:apple, color:red)
// 3. (name:lemon, color:yellow)
MyFruitCounter.Add(1); // Exported (No key/value pair)
MyFruitCounter.Add(1, new("name", "apple"), new("color", "red")); // Exported
MyFruitCounter.Add(2, new("name", "lemon"), new("color", "yellow")); // Exported
MyFruitCounter.Add(1, new("name", "lemon"), new("color", "yellow")); // Exported
MyFruitCounter.Add(2, new("name", "apple"), new("color", "green")); // Not exported
MyFruitCounter.Add(5, new("name", "apple"), new("color", "red")); // Exported
MyFruitCounter.Add(4, new("name", "lemon"), new("color", "yellow")); // Exported
// There are four distinct key/value combinations emitted for `AnotherFruitCounter`:
// 1. (name:kiwi)
// 2. (name:banana, color:yellow)
// 3. (name:mango, color:yellow)
// 4. (name:banana, color:green)
// Since the maximum number of `MetricPoint`s allowed is `3`, the SDK will only export measurements for the following three combinations:
// 1. No key/value pair (This is a special case. The SDK reserves a `MetricPoint` for it even if it's not explicitly emitted.)
// 2. (name:kiwi)
// 3. (name:banana, color:yellow)
AnotherFruitCounter.Add(4, new KeyValuePair<string, object>("name", "kiwi")); // Exported
AnotherFruitCounter.Add(1, new("name", "banana"), new("color", "yellow")); // Exported
AnotherFruitCounter.Add(2, new("name", "mango"), new("color", "yellow")); // Not exported
AnotherFruitCounter.Add(1, new("name", "mango"), new("color", "yellow")); // Not exported
AnotherFruitCounter.Add(2, new("name", "banana"), new("color", "green")); // Not exported
AnotherFruitCounter.Add(5, new("name", "banana"), new("color", "yellow")); // Exported
AnotherFruitCounter.Add(4, new("name", "mango"), new("color", "yellow")); // Not exported
```

To set the [cardinality limit](../README.md#cardinality-limits) at individual
metric level, use `MetricStreamConfiguration.CardinalityLimit`:
> `MetricStreamConfiguration.CardinalityLimit` is an experimental API only
available in pre-release builds. For details see:
[OTEL1003](../../diagnostics/experimental-apis/OTEL1003.md).

```csharp
var meterProvider = Sdk.CreateMeterProviderBuilder()
.AddMeter("MyCompany.MyProduct.MyLibrary")
.AddView(instrumentName: "MyFruitCounter", new MetricStreamConfiguration { CardinalityLimit = 10 })
// Set a custom CardinalityLimit (10) for "MyFruitCounter"
.AddView(
instrumentName: "MyFruitCounter",
new MetricStreamConfiguration { CardinalityLimit = 10 })
.AddConsoleExporter()
.Build();
```
Expand Down
8 changes: 6 additions & 2 deletions src/OpenTelemetry/CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,12 @@

* **Experimental (pre-release builds only):** Added support for setting
`CardinalityLimit` (the maximum number of data points allowed for a metric)
when configuring a view.
([#5312](https://github.com/open-telemetry/opentelemetry-dotnet/pull/5312))
when configuring a view (applies to individual metrics) and obsoleted
`MeterProviderBuilderExtensions.SetMaxMetricPointsPerMetricStream` (previously
applied to all metrics). The default cardinality limit for metrics remains at
`2000`.
([#5312](https://github.com/open-telemetry/opentelemetry-dotnet/pull/5312),
[#5328](https://github.com/open-telemetry/opentelemetry-dotnet/pull/5328))

* Updated `LogRecord` to keep `CategoryName` and `Logger` in sync when using the
experimental Log Bridge API.
Expand Down
46 changes: 23 additions & 23 deletions src/OpenTelemetry/Metrics/AggregatorStore.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ internal sealed class AggregatorStore
{
internal readonly bool OutputDelta;
internal readonly bool OutputDeltaWithUnusedMetricPointReclaimEnabled;
internal readonly int CardinalityLimit;
internal readonly bool EmitOverflowAttribute;
internal long DroppedMeasurements = 0;

private static readonly string MetricPointCapHitFixMessage = "Consider opting in for the experimental SDK feature to emit all the throttled metrics under the overflow attribute by setting env variable OTEL_DOTNET_EXPERIMENTAL_METRICS_EMIT_OVERFLOW_ATTRIBUTE = true. You could also modify instrumentation to reduce the number of unique key/value pair combinations. Or use Views to drop unwanted tags. Or use MeterProviderBuilder.SetMaxMetricPointsPerMetricStream to set higher limit.";
Expand Down Expand Up @@ -42,8 +44,6 @@ internal sealed class AggregatorStore
private readonly int exponentialHistogramMaxScale;
private readonly UpdateLongDelegate updateLongCallback;
private readonly UpdateDoubleDelegate updateDoubleCallback;
private readonly int maxMetricPoints;
private readonly bool emitOverflowAttribute;
private readonly ExemplarFilter exemplarFilter;
private readonly Func<KeyValuePair<string, object?>[], int, int> lookupAggregatorStore;

Expand All @@ -57,17 +57,17 @@ internal AggregatorStore(
MetricStreamIdentity metricStreamIdentity,
AggregationType aggType,
AggregationTemporality temporality,
int maxMetricPoints,
int cardinalityLimit,
bool emitOverflowAttribute,
bool shouldReclaimUnusedMetricPoints,
ExemplarFilter? exemplarFilter = null)
{
this.name = metricStreamIdentity.InstrumentName;
this.maxMetricPoints = maxMetricPoints;
this.CardinalityLimit = cardinalityLimit;

this.metricPointCapHitMessage = $"Maximum MetricPoints limit reached for this Metric stream. Configured limit: {this.maxMetricPoints}";
this.metricPoints = new MetricPoint[maxMetricPoints];
this.currentMetricPointBatch = new int[maxMetricPoints];
this.metricPointCapHitMessage = $"Maximum MetricPoints limit reached for this Metric stream. Configured limit: {this.CardinalityLimit}";
this.metricPoints = new MetricPoint[cardinalityLimit];
this.currentMetricPointBatch = new int[cardinalityLimit];
this.aggType = aggType;
this.OutputDelta = temporality == AggregationTemporality.Delta;
this.histogramBounds = metricStreamIdentity.HistogramBucketBounds ?? FindDefaultHistogramBounds(in metricStreamIdentity);
Expand All @@ -89,7 +89,7 @@ internal AggregatorStore(
this.tagsKeysInterestingCount = hs.Count;
}

this.emitOverflowAttribute = emitOverflowAttribute;
this.EmitOverflowAttribute = emitOverflowAttribute;

var reservedMetricPointsCount = 1;

Expand All @@ -105,17 +105,17 @@ internal AggregatorStore(

if (this.OutputDeltaWithUnusedMetricPointReclaimEnabled)
{
this.availableMetricPoints = new Queue<int>(maxMetricPoints - reservedMetricPointsCount);
this.availableMetricPoints = new Queue<int>(cardinalityLimit - reservedMetricPointsCount);

// There is no overload which only takes capacity as the parameter
// Using the DefaultConcurrencyLevel defined in the ConcurrentDictionary class: https://github.com/dotnet/runtime/blob/v7.0.5/src/libraries/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs#L2020
// We expect at the most (maxMetricPoints - reservedMetricPointsCount) * 2 entries- one for sorted and one for unsorted input
this.tagsToMetricPointIndexDictionaryDelta =
new ConcurrentDictionary<Tags, LookupData>(concurrencyLevel: Environment.ProcessorCount, capacity: (maxMetricPoints - reservedMetricPointsCount) * 2);
new ConcurrentDictionary<Tags, LookupData>(concurrencyLevel: Environment.ProcessorCount, capacity: (cardinalityLimit - reservedMetricPointsCount) * 2);

// Add all the indices except for the reserved ones to the queue so that threads have
// readily available access to these MetricPoints for their use.
for (int i = reservedMetricPointsCount; i < this.maxMetricPoints; i++)
for (int i = reservedMetricPointsCount; i < this.CardinalityLimit; i++)
{
this.availableMetricPoints.Enqueue(i);
}
Expand Down Expand Up @@ -164,12 +164,12 @@ internal int Snapshot()
}
else if (this.OutputDelta)
{
var indexSnapshot = Math.Min(this.metricPointIndex, this.maxMetricPoints - 1);
var indexSnapshot = Math.Min(this.metricPointIndex, this.CardinalityLimit - 1);
this.SnapshotDelta(indexSnapshot);
}
else
{
var indexSnapshot = Math.Min(this.metricPointIndex, this.maxMetricPoints - 1);
var indexSnapshot = Math.Min(this.metricPointIndex, this.CardinalityLimit - 1);
this.SnapshotCumulative(indexSnapshot);
}

Expand Down Expand Up @@ -227,7 +227,7 @@ internal void SnapshotDeltaWithMetricPointReclaim()

int startIndexForReclaimableMetricPoints = 1;

if (this.emitOverflowAttribute)
if (this.EmitOverflowAttribute)
{
startIndexForReclaimableMetricPoints = 2; // Index 0 and 1 are reserved for no tags and overflow

Expand All @@ -249,7 +249,7 @@ internal void SnapshotDeltaWithMetricPointReclaim()
}
}

for (int i = startIndexForReclaimableMetricPoints; i < this.maxMetricPoints; i++)
for (int i = startIndexForReclaimableMetricPoints; i < this.CardinalityLimit; i++)
{
ref var metricPoint = ref this.metricPoints[i];

Expand Down Expand Up @@ -440,7 +440,7 @@ private int LookupAggregatorStore(KeyValuePair<string, object?>[] tagKeysAndValu
if (!this.tagsToMetricPointIndexDictionary.TryGetValue(sortedTags, out aggregatorIndex))
{
aggregatorIndex = this.metricPointIndex;
if (aggregatorIndex >= this.maxMetricPoints)
if (aggregatorIndex >= this.CardinalityLimit)
{
// sorry! out of data points.
// TODO: Once we support cleanup of
Expand Down Expand Up @@ -469,7 +469,7 @@ private int LookupAggregatorStore(KeyValuePair<string, object?>[] tagKeysAndValu
if (!this.tagsToMetricPointIndexDictionary.TryGetValue(sortedTags, out aggregatorIndex))
{
aggregatorIndex = ++this.metricPointIndex;
if (aggregatorIndex >= this.maxMetricPoints)
if (aggregatorIndex >= this.CardinalityLimit)
{
// sorry! out of data points.
// TODO: Once we support cleanup of
Expand All @@ -496,7 +496,7 @@ private int LookupAggregatorStore(KeyValuePair<string, object?>[] tagKeysAndValu
{
// This else block is for tag length = 1
aggregatorIndex = this.metricPointIndex;
if (aggregatorIndex >= this.maxMetricPoints)
if (aggregatorIndex >= this.CardinalityLimit)
{
// sorry! out of data points.
// TODO: Once we support cleanup of
Expand All @@ -518,7 +518,7 @@ private int LookupAggregatorStore(KeyValuePair<string, object?>[] tagKeysAndValu
if (!this.tagsToMetricPointIndexDictionary.TryGetValue(givenTags, out aggregatorIndex))
{
aggregatorIndex = ++this.metricPointIndex;
if (aggregatorIndex >= this.maxMetricPoints)
if (aggregatorIndex >= this.CardinalityLimit)
{
// sorry! out of data points.
// TODO: Once we support cleanup of
Expand Down Expand Up @@ -929,7 +929,7 @@ private void UpdateLong(long value, ReadOnlySpan<KeyValuePair<string, object?>>
{
Interlocked.Increment(ref this.DroppedMeasurements);

if (this.emitOverflowAttribute)
if (this.EmitOverflowAttribute)
{
this.InitializeOverflowTagPointIfNotInitialized();
this.metricPoints[1].Update(value);
Expand Down Expand Up @@ -973,7 +973,7 @@ private void UpdateLongCustomTags(long value, ReadOnlySpan<KeyValuePair<string,
{
Interlocked.Increment(ref this.DroppedMeasurements);

if (this.emitOverflowAttribute)
if (this.EmitOverflowAttribute)
{
this.InitializeOverflowTagPointIfNotInitialized();
this.metricPoints[1].Update(value);
Expand Down Expand Up @@ -1017,7 +1017,7 @@ private void UpdateDouble(double value, ReadOnlySpan<KeyValuePair<string, object
{
Interlocked.Increment(ref this.DroppedMeasurements);

if (this.emitOverflowAttribute)
if (this.EmitOverflowAttribute)
{
this.InitializeOverflowTagPointIfNotInitialized();
this.metricPoints[1].Update(value);
Expand Down Expand Up @@ -1061,7 +1061,7 @@ private void UpdateDoubleCustomTags(double value, ReadOnlySpan<KeyValuePair<stri
{
Interlocked.Increment(ref this.DroppedMeasurements);

if (this.emitOverflowAttribute)
if (this.EmitOverflowAttribute)
{
this.InitializeOverflowTagPointIfNotInitialized();
this.metricPoints[1].Update(value);
Expand Down
Loading

0 comments on commit cf00e42

Please sign in to comment.