Messaging-System Metric semantic conventions

[cwildman]

Changes

Add semantic conventions for messaging system metrics.

Related issues:
Fixes #1014

PR Details

The general approach I took here was to take the messaging operations referenced in the tracing semantic conventions (sending, receiving, processing) and break them into 3 distinct metric groups. I created metrics for monitoring throughput and latency within these operations.

I've left individual commits to highlight some open questions/points of interest that I will elaborate on here.

Metric naming with Span Kind

@justinfoote mentioned that the preference is that when name spacing labels and metrics the second field should be the "span kind". The second commit attempts to follow this convention. However the span kind as defined in the tracing semantic conventions of messaging suggest producer/consumer should be used for async message processing, and server/client should be used for synchronous message processing. Should we change the metric names based on whether the application is sync or async? I assumed no. Also there's a bit of ambiguity here because the tracing semantic conventions yaml refers to things like messaging.consumer.synchronous instead of messaging.client.

Superfluous Counters vs. ease of use

I created a specific Counter to track the messages sent, received and processed. This information is already captured in the
count of the duration metric ValueRecorder. Personally I feel like the extra metric is worth the ease of use but I don't know where the OTEL community stands on this.

Semconv format and generator

I went ahead and attempted to use the semconv yaml format and generator. Many of these labels are identical to tracing messaging attributes but I went ahead and copied them into a separate yaml for messaging metrics. I could not get the generator to actually work though. Here's an example: docker run --rm otel/semconvgen --yaml-root ../../opentelemetry-specification/semantic_conventions markdown --markdown-root ../../opentelemetry-specification/specification

Batch metrics

The last commit adds metrics around batching because I see that as a common feature that needs to be monitored. Introducing metrics around batching does create confusion around whether some of the non-batch metrics still exist or what they mean. I'm interested to hear if others feel the behavior is sufficiently explained or if batch metrics should be dropped entirely.

Other metrics

Here's a list of other potential metric categories that I omitted. I'm assuming these are too specific to only a subset of messaging systems to be included here:

1. Lag
2. I/O time
3. Connection pooling
4. Errors
5. Retries

@jmacd @bogdandrutu mentioning you as potential reviewers.

[linux-foundation-easycla]

The committers are authorized under a signed CLA.

• ✅ cwildman (53fd5a5, 9adc80e, deb28ab, 9f4a7b5, 6dbd507, ba169fe, a60bc2d, 612115e, 03cb521, 8c0d5b2, 183816a)

[arminru]

Please add an entry in CHANGELOG.md

[kenfinnigan]

Based on the work on #1027, the net.* names should be removed as constraints and added as references like for tracing

[kenfinnigan]

The metric name might need altering, or messaging.consumer.messages needs altering to include "receive" in the name.

Otherwise, this metric could be implied to be a child of the other metric by virtue of the naming.

[cwildman]

That makes sense, I'll think about how to reword these. My instinct was to have the metric name be messaging.processed.messages or messaging.processor.messages, however following the span kind naming convention it says that the processor should use "consumer" as its span kind.

[cwildman]

I was able to resolve some of my issues with the semconv generator. There were a few things going on:

1. My /Users bind mount in docker for mac doesn't work even though it is automatically included.
2. The semconv generator doesn't allow for multiple instances of the same id, e.g. messaging. I just used a different id for the metrics messaging labels named metrics-messaging.
3. I used a reference to point to all the labels that would have the same name as an existing tracing attribute.

While the generator works correctly for me it seems I'm failing the md-check with File /spec/metrics/semantic_conventions/messaging.md contains a table that would be reformatted. even though I'm using the same generator to create the table. Anyone have suggestions for that?

[jmacd]

Discussed in the Metrics SIG today: we think it would be best to eliminate the superfluous counters. This may be problematic in the short term, but in the long term we can address these with a Views API, we think.

[cwildman]

@jmacd thanks for the heads up. Removed those.

[cwildman]

@open-telemetry/specs-metrics-approvers this is ready for review!

[justinfoote]

@open-telemetry/specs-metrics-approvers this is ready for review!

[MrAlias]

The structure, labels, and instrument definitions look good. Just issues with some of the terminology and formatting. Good job 👍

[andrewhsu]

at the metrics sig mtg today talked about this issue's staleness

[cwildman]

@MrAlias I think I addressed almost all of your feedback. I had questions/comments on a couple of them. Thanks!

[jmacd]

I re-read this PR and wonder if the recently-added guidelines on metrics named .io should be applied here. Instead of having two metrics <something>.bytes for each direction of travel, we can use <something>.io with a label to indicate whether it was produced or consumed.

[cwildman]

@jmacd interesting. To be a little more concrete you're imagining renaming these metrics:

messaging.producer.bytes -> messaging.io
messaging.consumer.received.bytes -> messaging.io

And then we'd have some labels like direction/operation which can be one of ['produce', 'consume']?

Some of my initial reactions to this:

1. What happens to the other metrics like messaging.producer.duration? It seems confusing as a user that there's not a consistent naming scheme for my producer metrics.
2. I'm surprised the semantic conventions recommend using .io for any bidirectional data flow. For me I/O is specific to data flowing in or out of process from external devices. This is in contrast to data flow in process that is abstracted from the raw I/O that may have been involved or may not have come directly from I/O at all.
3. It seems like the .io convention should have some guidance on what the direction label should be called.

[aabmass]

@cwildman maybe this could be helpful to decide:

As a rule of thumb, aggregations over all the dimensions of a given metric SHOULD be meaningful," as Prometheus recommends.

Would a user ever want to aggregate together the produced/consumed bytes into one timeseries (or would it be meaningful)? If not, then I suppose they should be separate.

It seems like the .io convention should have some guidance on what the direction label should be called.
That should probably be specified. For system metrics, it's label direction with value read | write

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

[jgals]

Hey @cwildman and @jmacd, I'm commenting here to avoid having this closed by the bot. 🤖

[cwildman]

I'd like to reopen this PR and hopefully discuss at the spec meeting this week.

[sergeykonkin]

@cwildman Hi!
Any chances to revive this PR and finish a work on this? We are looking forward on this specification