EC2 Service Discovery Filter Not Working Correctly

[james-miller]

Proposal

Use case. Why is this important?
I’m looking to use AWS EC2 tags to indicate which exporters are installed and target the correct ports. This will allow us to easily enable new metrics and standardise our Prometheus configuration using a tag like "PrometheusMonitoring".

The documentation suggests that this is currently possible using filters but this does not seem to function as intended.

Bug Report

What did you do?
Added the following job to my scrape config:

  - job_name: 'linux'

    ec2_sd_configs:
      - region: 'eu-west-1'
      - port: 9100
      - filters:
          - name: tag:PrometheusMonitoring
            values:
              - '*Node*'
              - '*node*'

    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: Hostname
      - source_labels: [__meta_ec2_tag_CustomerID]
        target_label: CustomerID
      - source_labels: [__meta_ec2_tag_Environment]
        target_label: Environment
      - source_labels: [__meta_ec2_instance_id]
        target_label: InstanceID

What did you expect to see?
One active target to be displayed in the 'Linux' job on the Service Discovery page. This should be just the instance with the 'PrometheusMonitoring' tag key and 'SQL, Node, Squid' in the tag value.

What did you see instead? Under which circumstances?
All of the instances in the AWS account were picked up twice by that job on the service discovery page.

Also worth noting that the following AWS command works perfectly:

aws ec2 describe-instances --region eu-west-1 --filters 'Name=tag:PrometheusMonitoring,Values="*Node*","*node*"'

Environment

• System information:

  Linux 3.10.0-862.14.4.el7.x86_64 x86_64
  Docker version 18.09.1, build 4c52b90

• Prometheus version:

  prom/prometheus:v2.7.1

• Prometheus configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

  external_labels:
      region: 'eu-west-1'
      environment: '**'

rule_files:
 - rules.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']


  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']


  - job_name: 'windows'
    ec2_sd_configs:
      - region: 'eu-west-1'
        port: 9182
        filters:
          - name: platform
            values:
              - windows
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: Hostname
      - source_labels: [__meta_ec2_tag_CustomerID]
        target_label: CustomerID
      - source_labels: [__meta_ec2_tag_Environment]
        target_label: Environment
      - source_labels: [__meta_ec2_instance_id]
        target_label: InstanceID


  - job_name: 'linux'
     ec2_sd_configs:
      - region: 'eu-west-1'
      - port: 9100
      - filters:
          - name: tag:PrometheusMonitoring
            values:
              - '*Node*'
              - '*node*'
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: Hostname
      - source_labels: [__meta_ec2_tag_CustomerID]
        target_label: CustomerID
      - source_labels: [__meta_ec2_tag_Environment]
        target_label: Environment
      - source_labels: [__meta_ec2_instance_id]
        target_label: InstanceID

• Service Discovery Page:
  

[simonpasquier]

Which version of Prometheus do you run? Can you share a screenshot of the service discovery page?
I looked quickly at the SD code and it seems ok but I don't have any EC2 account to test...

[james-miller]

We're running v2.7.1 of Prometheus.
I've updated the original post with the screenshot.
There are currently only 21 instances in that AWS account (of which, only 1 has the applicable tag attached) so it must not be filtering correctly and its duplicating all of the instances it is discovering.

[simonpasquier]

Is your linux job really formatted like this?

  - job_name: 'linux'
    ec2_sd_configs:
      - region: 'eu-west-1'
      - port: 9100
      - filters:
          - name: tag:PrometheusMonitoring
            values:
              - '*Node*'
              - '*node*'

If yes, it creates 3 SD instances and I suspect that you want this instead:

  - job_name: 'linux'
    ec2_sd_configs:
      - region: 'eu-west-1'
        port: 9100
        filters:
          - name: tag:PrometheusMonitoring
            values:
              - '*Node*'
              - '*node*'

[james-miller]

That 's fixed it.
Thanks for your help

[brian-brazil]

Hmm, we're meant to de-duplicate identical instances post relabelling within a scrape config. Was that not happening?

[james-miller]

All of the instances in that job were listed twice, so I don't believe it was here

[brian-brazil]

Actually one set would have had port 9100, and two sets 80 so 2x is right.