Counter generated from System.Diagnostic.Metrics resets to zero.

[bthharper]

Description

The counters in .NET 8 (or .NET 4.8) reset to zero whilst running, losing the current measurement.

Reproduction Steps

Run the following code sample in Rider (or on the command line) ..

using System.Diagnostics.Metrics;

Console.WriteLine("Hello, World!");

using var meter = new Meter("Metrics", "1.0.0");

var counter1 = meter.CreateCounter<long>("1.count");
var counter2 = meter.CreateCounter<long>("2.count");
var counter3 = meter.CreateCounter<long>("3.count");
var counter4 = meter.CreateCounter<long>("4.count");

using var meterListener = new MeterListener();

meterListener.SetMeasurementEventCallback<long>(ShowMeasurement);
meterListener.EnableMeasurementEvents(counter1);
meterListener.EnableMeasurementEvents(counter2);
meterListener.EnableMeasurementEvents(counter3);
meterListener.EnableMeasurementEvents(counter4);

var rnd = Random.Shared;
while (true) {
    Thread.Sleep(TimeSpan.FromSeconds(rnd.Next(0, 5)));
    switch (rnd.Next(0, 4)) {
        case 0:
            counter1.Add(1);
            break;
        case 1:
            counter2.Add(1);
            break;
        case 2:
            counter3.Add(1);
            break;
        case 3:
            counter4.Add(1);
            break;
    }
}

static void ShowMeasurement<T>(
    Instrument instrument,
    T measurement,
    ReadOnlySpan<KeyValuePair<string, object>> tags,
    object state) {
    Console.WriteLine($"{instrument.Name} = {measurement}");
}

Expected behavior

The counters to increment until the application is stopped.

Actual behavior

Regression?

No response

Known Workarounds

No response

Configuration

.NET 8 (8.0.5) and .NET 4.8.1
Windows 11 (latest)
x64

Other information

No response

[tarekgh]

This is not how the metrics work in .NET. It is the listener/aggregator job to accumulate the published counter values. I encourage you to read the doc https://learn.microsoft.com/en-us/dotnet/core/diagnostics/metrics-collection#create-a-custom-collection-tool-using-the-net-meterlistener-api for more info and explanation.

[julealgon]

It is the listener/aggregator job to accumulate the published counter values.

@tarekgh do you mind if I ask a related question to this? I've never found proper documentation about it.

In a setup with an OTEL collector and an observability tool (e.g. Datadog), who exactly is performing this aggregation? Is it the application itself before emitting the values to the collector? Is it the collector? Or does the collector also sends raw values and they are aggregated by Datadog?

This has always confused me because it is not clear to me when a counter is supposed to reset, and how one even signals an intended reset (if that is even possible). Does a reset happen when I restart my application? Is there some sort of signal that is sent from the application side to indicate a counter is "a new instance" or "a new stream of deltas separate from previous ones", or is that defined by the collector or the observability tool?

Since counters are delta based, one would need to accumulate their values forever to have an accurate view: basically going back to the very first measurement. For example, imagine I wanted to track "hats sold throughout history", and my application has been running for 10 years. Do telemetry counters (or any other instruments really) work at that scale, or are they not meant to capture "persistent" telemetry like this?

I probably just missed it, but I couldn't find the answer for this "delta instrument aggregation lifetime" question in either the Microsoft docs, OTEL docs or even Datadog docs.

[tarekgh]

do you mind if I ask a related question to this?

You are welcome to ask any question anytime.

There is a good doc https://github.com/open-telemetry/opentelemetry-dotnet/tree/4af3df9b81dbf78f3abbbd5d764573db8d54d373/docs/metrics#meterprovider-management describing how metrics work in OpenTelelmetry. Please have a look and let's know if you still have any questions not answered.

CC @reyang @CodeBlanch @noahfalk