Gauge: arithmetic operations on two unix timestamps lead to empty time series

[shatting]

Tried subtracting (but the same holds for *,/,+) two unix timestamps from the same job/instance. The /metrics endpoint shows them as:

gauge{country="<country>",exported_country="<country>",instance="<instance>",
job="<job>",quantity="unix_timestamp1"} 1469875400237
gauge{country="<country>",exported_country="<country>",instance="<instance>",
job="<job>",quantity="unix_timestamp2"} 1469875389324

In the prometheus GUI, i can evaluate each metric and get a value, and i can graph:

Element: gauge{country="<country>",exported_country="<country>",instance="<instance>",
job="<job>",quantity="unix_timestamp1"} 
Value: 1469875400237

However, trying to subtract the two timestamps (in either order)

gauge{country="<country>",exported_country="<country>",instance="<instance>",
job="<job>",quantity="unix_timestamp1"} 
- 
gauge{country="<country>",exported_country="<country>",instance="<instance>",
job="<job>",quantity="unix_timestamp2"}

leads to an empty time series

Element: no data
Value:

Subtracting a timestamp from itself

gauge{country="<country>",exported_country="<country>",instance="<instance>",
job="<job>",quantity="unix_timestamp1"} 
- 
gauge{country="<country>",exported_country="<country>",instance="<instance>",
job="<job>",quantity="unix_timestamp1"}

leads to the expected

Element: gauge{country="<country>",exported_country="<country>",instance="<instance>",
job="<job>",quantity="unix_timestamp1"}
Value: 0

The only anomaly i could see causing this behaviour is the exported_country label which comes from the same label name being used in the client and the scrape config.

Environment

• System information:

  FreeBSD 10.2-RELEASE-p9 amd64

• Prometheus version:

  prometheus, version 1.0.1 (branch: master, revision: be40190)
  build user: root@e881b289ce76
  build date: 20160722-20:02:55
  go version: go1.6.2

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  # - "first.rules"
  # - "second.rules"


scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5m

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9091']

  -..<other jobs>..

  - job_name: esSync
    scrape_interval: 5m
    static_configs:
      - targets: ['instance1']
        labels:
          country: country1
      - targets: ['instance2']
        labels:
          country: country2

[shatting]

classic case of RTFM
metric1 - ignoring (quantity) metric2
fixes the "issue".

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.