Open telemetry .NET runtime instrumentation overhead

[niroshikaias-hub]

Hi team,
I’m working with OpenTelemetry .NET (v1.15.x) using runtime instrumentation + Prometheus exporter, and I’m seeing continuous memory growth even when my service is mostly idle.
Setup
Using:
AddRuntimeInstrumentation()
AddPrometheusExporter()
Removed:
AddAspNetCoreInstrumentation()
Prometheus scrape interval: 30s
Added:
SetMaxMetricStreams(200)
My Code snippet:

builder.Services.AddOpenTelemetry()
.ConfigureResource(resource => resource.AddService("Service"))
.WithMetrics(metrics =>
{
metrics.AddRuntimeInstrumentation();
metrics.SetMaxMetricStreams(200);
metrics.AddPrometheusExporter();
});

app = builder.Build();
app.MapPrometheusScrapingEndpoint();

Observations:
Service is mostly idle (very low request rate)
But I still see:
Continuous allocation spikes
LOH (Large Object Heap) growing step-by-step
GC is running, but heap is not dropping
Heap (Gen2 + LOH) keeps increasing over time

What I tested:
Removed ASP.NET instrumentation → no major change
Tried AddView filtering → either no improvement or metrics dropped when AddView.Drop is added after adding few AddView(gc,heap,loh)
Prometheus scrape already reduced (30s)
My query is:
Is this expected behavior of:
runtime instrumentation + metrics SDK buffering/exporter, or
could this indicate a memory retention issue in OTel metrics pipeline?
Also,is there any recommended way to:
reduce memory footprint of runtime metrics?
control retention/allocation behavior?
or is this overhead expected when using runtime metrics + Prometheus exporter?
My main goal is to see GC,Heap,LOH,POH of the service(which runs in .net core).
Note: I cannot try OTEL zero instrumentation because my service code already have dynatrace logging as a profiler.
Thanks in advance!

[martincostello]

I wouldn't expect to see memory usage grow indefinitely with stable usage, no.

If you can create a public GitHub repository containing app that's a minimal repro of just enough to show what you're using, we could take a look and see if there's an issues cause by the OTel libraries (assuming no bugs in your app).

[niroshikaias-hub]

Thanks for the response.

I created a minimal repro using OpenTelemetry .NET (runtime metrics + Prometheus exporter), and I do not observe any memory leak there. Memory increases under load and is reclaimed correctly after GC, so the baseline OTel setup looks healthy.

However, in our actual service, we are seeing sustained memory growth (especially in LOH and total heap) during real traffic, even after GC activity. The current setup in our service mainly uses RuntimeInstrumentation and Prometheus exporter (ASP.NET instrumentation is currently disabled).

This suggests the issue might not be a direct OTel bug in isolation, but possibly related to:

• higher traffic patterns / concurrency
• interaction between runtime metrics and exporter
• or application-specific behavior under load

I will work on creating a minimal public repro that more closely simulates our real traffic pattern and service behavior, and share it here.

Meanwhile, if there are any known cases where runtime instrumentation or Prometheus exporter could lead to increased memory usage under high load, please let me know.

Thanks!

[martincostello]

Could you maybe be hitting this issue with Kestrel's memory pool not shrinking that was fixed in .NET 10? dotnet/aspnetcore#27394