Enable injecting component-specific Prometheus labels

[mmanciop]

Support static Prometheus labels in Magma

Problem

In the Prometheus ecosystem, "static" labels (i.e., labels that apply to all metrics retrieved from one endpoint) are best configured at target level in the scraper (e.g., the Prometheus server). This is how, for example, one easily labels all metrics coming from one exporter as environment=prod. Given this best practice, the Go Prometheus client does not provide facilities to add static labels.

Unfortunately, due to the way metrics are moved from single services to Metricsd and then to EventHub, it is not possible to specify static labels for a single service. This is something we really need in the charming of Magma with Juju, so that we can apply Juju topology to the metrics emitted by the various Magma services.

Solution

Ideally, we would like to be able to specify an environment variable called PROMETHEUS_LABELS (or something to that extent) that, when found by a Magma service, makes is automatically add the comma-separated key-value pairs of the Prometheus labels to the metrics data it sends to Metricsd.

We are not sure, however, on what would be the best way to introduce support for PROMETHEUS_LABELS. It seems like the NewWWWExporter could be the place where do build this, but it seems ungainly.

Non-goals

Anything specifically excluded from the concept

[hcgatewood]

Excited for this!

As far as implementation, I think the analytics service is not the ideal spot. Here's what I'm thinking instead — the metricsd.GetMetrics function [0]

This function is called from the GetMetrics method [1] of our Service303 gRPC servicer, which runs on every Orc8r service

I think the best way to hook this up is 1. add some code in the service init method [2] which reads in the desired env vars and places them into the returned Service object, then 2. accessing those values in the GetMetrics grpc method and passing them into the metricsd.GetMetrics function

[0] https://sourcegraph.com/github.com/hcgatewood/magma/-/blob/orc8r/lib/go/metrics/metrics_export.go#L67:1-88:1
[1] https://sourcegraph.com/github.com/hcgatewood/magma/-/blob/orc8r/lib/go/service/service303.go#L46:1-56:1
[2] https://sourcegraph.com/github.com/hcgatewood/magma/-/blob/orc8r/lib/go/service/service.go#L96:1-140:1

[hcgatewood]

Closing as accepted