[feature] Ability to specify consul_sd service with tag

[MrMMorris]

Would it be possible to have the ability to specify tags along with consul services? something like:

scrape_configs:
  - job_name: 'node'

    consul_sd_configs:
      - server: 'consul.service.consul:8500'
        datacenter: 'ec2'

        services:
          - 'monitoring:node'

scrape_configs:
  - job_name: 'container'

    consul_sd_configs:
      - server: 'consul.service.consul:8500'
        datacenter: 'ec2'

        services:
          - 'monitoring:container'

As opposed to having to have separate service names like this:

scrape_configs:
  - job_name: 'node'

    consul_sd_configs:
      - server: 'consul.service.consul:8500'
        datacenter: 'ec2'

        services:
          - 'monitoring-node'

scrape_configs:
  - job_name: 'container'

    consul_sd_configs:
      - server: 'consul.service.consul:8500'
        datacenter: 'ec2'

        services:
          - 'monitoring-container'

Or maybe

services:
  - 'monitoring'

tags: 
  - 'node'

Does this make sense?

[MrMMorris]

Or somewhat related: where do consul tags show up? there is tag_separator in the config, but I haven't seen them anywhere in the metadata labels

[fabxc]

The Consul tags are in the __meta_consul_tags label. With a relabeling rule you can filter the targets you want to scrape based on whether 'container' or 'node' show up in there.

scrape_configs:
- job_name: 'monitoring'

  consul_sd_configs:
  - server: 'consul.service.consul:8500'
    datacenter: 'ec2'
    services:
    - 'monitoring'

  relabel_configs:
  - source_labels: ['__meta_consul_tags']
    regex: '(container|node)`
    action: keep

In general you don't want to open two entirely new scrape configs to scrape targets from two services.
This is way shorter unless if you want to use different scrape options for both:

scrape_configs:
- job_name: 'node'

  consul_sd_configs:
  - server: 'consul.service.consul:8500'
    datacenter: 'ec2'

    services: ['service-a', 'service-b']

[MrMMorris]

amazing! thanks.

Two answers in one 😉

[joonas-fi]

Sorry for digging up old graves, but I just wanted to clarify something that I struggled with.

Thanks @fabxc for the tip on using relabel_configs with __meta_consul_tags and regex! I wouldn't have guessed that you can use that as a filter for targets. In fact, I think this is super confusing, as even the docs say:

Relabeling is a powerful tool to dynamically rewrite the label set of a target before it gets scraped.

Nothing suggests to me that this system can be used to filter out target candidates advertised by the service discovery backend.

Anyways, using regex has_metrics_endpoint didn't work for me at all. I lost quite some time until I read from the docs:

The regex is fully anchored.

Which I didn't immediately understand to mean that it essentially makes my regex to mean ^has_metrics_endpoint$ which I guess is nice in some cases, but in this case this was magic that totally tripped me off. I'd expect myself to explicitly decide on how the regex works, instead of tripping on magical behaviour.

I'm glad I accidentally hovered my mouse over "Labels" cell in Prometheus' UI (/targets) and noticed that __meta_consul_tags=,has_metrics_endpoint, so I tuned the regex to be .*,has_metrics_endpoint,.*and Bob's my uncle. :)

Anyways, I'm thankful for @fabxc's suggestion as my problem is now solved. :)

edit: What I'm trying to say is that the docs should mention on how to filter targets by tag, as I guess that is what almost everybody has to do anyways, since not all Consul-advertised services have a /metric endpoint and so said targets would be treated as errors.

[fabxc]

Thanks for the feedback, we should probably make our documentation clearer.

On Thu, Jun 9, 2016, 6:08 PM joonas-fi notifications@github.com wrote:

Sorry for digging up old graves, but I just wanted to clarify something
that I struggled with.

Thanks @fabxc https://github.com/fabxc for the tip on using
relabel_configs with __meta_consul_tags and regex! I wouldn't have guessed
that you can use that as a filter for targets. In fact, I think this is
super confusing, as even the docs say:

Relabeling is a powerful tool to dynamically rewrite the label set of a
target before it gets scraped.

Nothing suggests to me that this system can be used to filter out target
candidates advertised by the service discovery backend.

Anyways, using regex has_metrics_endpoint didn't work for me at all. I
lost quite some time until I read from the docs:

The regex is fully anchored.

Which I didn't immediately understand to mean that it essentially makes my
regex to mean ^has_metrics_endpoint$ which I guess is nice in some cases,
but in this case this was magic that totally tripped me off. I'd expect
myself to explicitly decide on how the regex works, instead of tripping on
magical behaviour.

I'm glad I accidentally hovered my mouse over "Labels" cell in Prometheus'
UI (/targets) and noticed that __meta_consul_tags=,has_metrics_endpoint,
so I tuned the regex to be ._,has_metrics_endpoint,._and Bob's my uncle.
:)

Anyways, I'm thankful for @fabxc https://github.com/fabxc's suggestion
as my problem is now solved. :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#771 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AEuA8vDfoFEI1nFtEClOng_71KAPA3p8ks5qKDqUgaJpZM4E3oSO
.

[sbilello]

Let's suppose you have a service A tagged with my-test-service and it has been discovered via consul and used as target from Prometheus.
If the service now it is ready for production and you want to re-tag it with service-production-ready how can you do that?
I would like to set up some alerts only on production machines with a particular tag and I don't want that the alerts will evaluate test machines.
Thank you in advance for your help!