Add Consul Tags to labels

[lswith]

Consul allows for tags to be added to a service. I'd like to be able to import these into the labels for a metric.

[grobie]

Sounds good. I'd implement this as separate metric and use joins. E.g.

consul_service_tags{service_id="xyc", tags=",foo,bar,baz,"} 1

[lswith]

I was more thinking along the lines of:

consul_service_health_status{service_id:'xyz', service_tags:'primary,master,env1'}

Relabelling could then be applied to the 'tags' label and allow for more advanced labels to be added if desired.

I'd put the service_tags label wherever the service_id is present.

[grobie]

No, this label shouldn't be added to all metrics:

• Joins work fine here, it's not sane to replicate the same information over and over
• Tags are not necessarily immutable, which would create new time series every time any label gets updated
• Labels come at a cost, the more are added the more elaborate aggregations become. As not every query requires a tag label matcher, I don't agree to make such queries harder.

On another note, lists of tags should start and end with the same delimiter (a comma probably), so that regular expressions become easier. As an exampple, {tags=~".*,foo,.*"} is now possible without the need to add 4 variants with/without leading/trailing commas.

[ribbybibby]

I would find the addition of tags to existing service metrics useful to my use-case, although possibly I'm just not familiar enough with the query language to accomplish it with a separate metric.

I'd like to alert when a health status on a particular services goes warning or critical but direct the course of the alert based on labels created from the tags ({env="live"} etc). I'm not sure how I could accomplish this without having the tags as part of the health metric.

[dhiren051]

I am using consul exporter with source_labels: address and target_label: instance. I do see consul_health_service_status metrics but not consul_service_tags. How do I get consul tag using consul exporter ? Also , is there a way to filter tags using consul exporter similar to consul_sd_config's __meta_consul_tag relabel ?

[peimanja]

any updates on this one?

[jeinwag]

@peimanja there is the consul_service_tag metric which exports the tags.

[peimanja]

@jeinwag I already noticed that we have consul_service_tag as a separate metric but I am looking for something more like the one @lswith is suggesting

consul_service_health_status{service_id:'xyz', service_tags:'primary,master,env1'}

[ribbybibby]

I'm with @peimanja in the sense that in a perfect world I'd like to be able to join tags to consul_service_health_status in such a way that I end up with this:
{node="web-01",service_id="apache",status="critical",tags=",prod,dc-1,teamA,"} 1

This would allow me to define a fairly generic alert rule and then route based on tags in Alertmanager. If this is possible I would love to hear about it but I can't seem to get there with metric relabelling and/or joining with PromQL.

The best I've managed to accomplish is to select series based on a single label match:
max(consul_health_service_status{status=~"(critical|warning)"}) by (node,service_id,status) * on (service_id,node) group_left (tag) consul_service_tag{tag="prod"} == 1

Which will allow me to select services tagged 'prod' which are critical or warning:
{node="web-01",service_id="apache",status="critical",tag="prod"} 1

[peimanja]

Any update on this? this could really simplify our queries and @ribbybibby workaround works fine but it makes it hard to write alert rules for different environments and use other tags to route alerts.

[debeste123]

The current setup with tags separated in metric "consul_service_tag" is not sufficient. People add multiple tags to one service, and being able to group_left on one of them does not add a lot of value.

• Giving all tags of a service with an alert is now not possible, only one.
• Creating alert based on the value of two tags of a service is not possible.

I don't want to sound ungrateful, because this is a great tool that I can use for free, but this should be implemented as @lswith suggests.
If you worry about performance and size used in the DB, you can hide this option behind a Flag.
So only idiots (like me) that care about functionality can use it.

[debeste123]

Telegraf handles tags exactly like I'm suggesting, so I will be moving to that solution.
Telegraf has a consul.input and a prometheus.output plugin, works just fine and is extremely easy to use.

[f0def]

Any updates on this issue?