Alerting time series misses external label

[svenmueller]

What did you do?
Querying for alert time series data.

What did you expect to see?
The labels should be the same for the same alert timeseries data.

What did you see instead? Under which circumstances?

When querying for alerts, two timeseries are returned instead of a single one. One is missing the external label "hostname".

Example:
Alert "InstanceLowMemory" is fired for instance "domain.com" which creates a timeseries with external label "hostname". But for the same alert, there is also a different timeseries without label "hostname".

Query:

ALERTS{alertstate="firing", customer="abc", alertname="InstanceLowMemory"} 

Result:

ALERTS{alertname="InstanceLowMemory",alertstate="firing",customer="abc",environment="stage",hostname="prometheus-yalow",instance="domain.com",job="node",region="eu",severity="page"}
ALERTS{alertname="InstanceLowMemory",alertstate="firing",customer="abc",environment="stage",instance="domain.com",job="node",region="eu",severity="page"}

Environment

• System information:

  Linux 3.13.0-105-generic x86_64

• Prometheus version:

  prometheus, version 1.4.1 (branch: master, revision: 2a89e87)
  build user: root@e685d23d8809
  build date: 20161128-09:59:22
  go version: go1.7.3

• Alertmanager version:

  alertmanager, version 0.5.1 (branch: master, revision: 0ea1cac51e6a620ec09d053f0484b97932b5c902)
  build user: root@fb407787b8bf
  build date: 20161125-08:14:40
  go version: go1.7.3

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      hostname: 'prometheus-awesome-yalow'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - "/etc/prometheus/rules/*.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']


  # Scrape the Node Exporter every 5 seconds.
  - job_name: 'node'
    scrape_interval: 5s
    file_sd_configs:
      - files:
        - /etc/prometheus/targets/*.yaml

    relabel_configs:
      - source_labels: [__address__]
        regex: (.*):9100
        replacement: ${1}
        target_label: instance
      - source_labels: [instance]
        regex: .*\.([\w,-]+)\.domain\.com
        replacement: ${1}
        target_label: region
      - source_labels: [instance]
        regex: (\w+).*
        replacement: ${1}
        target_label: customer
      - source_labels: [instance]
        regex: .*-(dev|stage|prod)-.*
        replacement: ${1}
        target_label: environment

alerting:
  alertmanagers:
    - static_configs:
      - targets: ["alertmanager:9093"]

[brancz]

My guess is that this is happening because external labels are only attached at the point when a notification is sent out, whereas we should have already added them at the point where it is added to the storage. Does that sounds right @brian-brazil @fabxc ?

[brian-brazil]

They should only be added when they're sent out, otherwise would not be only external labels.

[brancz]

Makes sense. So which time-series should exist then/not exist @brian-brazil ?

[brian-brazil]

Only the timeseries without external labels should exist.

[brancz]

Ok that makes sense as well. I'll have a look later if I can find something that could be causing this.

[fabxc]

@brancz did you look into that any further?

[brancz]

I don't think so - sorry. I don't think I found what was causing this, but I also don't remember how much time I spent on it.

[brian-brazil]

This was fixed.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.