This is actually a dotnet/runtime problem (in my opinion), but it definitely breaks dotnet-counters so I figured I'd start here.
When a monitored application emits a System.Diagnostics.Metrics instrument whose tag values contain , or =, dotnet-counters displays the counter name but never renders any tag columns or values, and the display stops updating.
Although it's a bug in counters, the underlying problem in the runtime is that MetricsEventSource flattens the tag list for out-of-process consumers into a key=value,key=value string, which can't be reliably parsed if value contains commas or equal signs.
Minimal .NET 10 repro:
using System.Diagnostics.Metrics;
using var meter = new Meter("TagTest");
var counter = meter.CreateCounter<long>("hits");
Console.WriteLine($"PID: {Environment.ProcessId}");
while (true)
{
counter.Add(1,
new KeyValuePair<string, object?>("plain", "simple"),
new KeyValuePair<string, object?>("comma", "a,b,c"),
new KeyValuePair<string, object?>("equals", "x=1"),
new KeyValuePair<string, object?>("url", "/api/items?filter=red,blue&sort=name"));
Thread.Sleep(250);
}
The payload looks like this, and the problem is already obvious:
comma=a,b,c,equals=x=1,plain=simple,url=/api/items?filter=red,blue&sort=name
When you run dotnet-counters monitor --process-id <PID> --counters TagTest, instead of columns for plain, comma, equals, and url, you get:
...then it stops updating. I didn't dig into dotnet-counters to find that problem because I had already run into the general tagging problem in my own program, and I wondered if counters had some magic to fix it. (I did actually come up with some ugly code that resolves those, but a tag like key=a=1,b=2 where a=1,b=2 is the value can't be resolved; definitely an edge case but still a problem.)
I suppose some kind of escaping of commas and equal signs in the key values is called for, or emit the tag list in a structured fashion (JSON, a Dictionary, etc). ... obviously that's up to you folks. As far as I know OTel doesn't require any particular format, so there wouldn't be a violation of any standard if this changes.
Environment:
- dotnet-counters 9.0.661903
- .NET SDK 10.0.301
- Linux x64
This is actually a dotnet/runtime problem (in my opinion), but it definitely breaks
dotnet-countersso I figured I'd start here.When a monitored application emits a
System.Diagnostics.Metricsinstrument whose tag values contain,or=,dotnet-countersdisplays the counter name but never renders any tag columns or values, and the display stops updating.Although it's a bug in counters, the underlying problem in the runtime is that
MetricsEventSourceflattens the tag list for out-of-process consumers into akey=value,key=valuestring, which can't be reliably parsed ifvaluecontains commas or equal signs.Minimal .NET 10 repro:
The payload looks like this, and the problem is already obvious:
When you run
dotnet-counters monitor --process-id <PID> --counters TagTest, instead of columns forplain,comma,equals, andurl, you get:...then it stops updating. I didn't dig into dotnet-counters to find that problem because I had already run into the general tagging problem in my own program, and I wondered if counters had some magic to fix it. (I did actually come up with some ugly code that resolves those, but a tag like
key=a=1,b=2wherea=1,b=2is the value can't be resolved; definitely an edge case but still a problem.)I suppose some kind of escaping of commas and equal signs in the key values is called for, or emit the tag list in a structured fashion (JSON, a
Dictionary, etc). ... obviously that's up to you folks. As far as I know OTel doesn't require any particular format, so there wouldn't be a violation of any standard if this changes.Environment: