Remote writes not sending `job` label after the first few writes

[ghost]

What did you do?
Configured prometheus to write to a remote endpoint

What did you expect to see?
A constant flow of requests to that endpoint including a job label

What did you see instead? Under which circumstances?
The requests following the first few requests did not include job. The change occurs fairly quickly in a fresh/plain install where the only configuration is setting a remote write URL.

Environment
Mac OSX Sierra 10.12.6 and Centos RHEL 7

• System information:

  Darwin 16.7.0 x86_64 and Linux 3.10.0-514.26.2.el7.YAHOO.20170707.8.x86_64 x86_64

• Prometheus version:

prometheus, version 2.0.0-beta.1 (branch: HEAD, revision: 4dcb465)
build user: root@d3f9974fac5a
build date: 20170811-12:16:25
go version: go1.8.3

• Prometheus configuration file:

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']
remote_write:
  - url: http://localhost:1337/write

[ghost]

It seems to be the case that when a metric is cached, the "rule labels" are not appended, i.e. the behaviour that ruleLabelsAppender offers does not occur in the case where the metric is cached (by cached, I mean when the conditional statement on line 776 of retrieval/scrape.go is true).

https://github.com/prometheus/prometheus/blob/dev-2.0/retrieval/scrape.go#L776

[brian-brazil]

@tomwilkie

[ghost]

A possibly related issue is the caching dropping certain metrics and not just certain labels. When I disabled caching (by setting ok = false), I noticed a certain metric that was only showing up in the first few writes now showing up consistently among the writes.

The change goes into effect at 15:30, for reference. Each of the previous spikes is from the initial writes, after which the metrics just stop being reported.

[krasi-georgiev]

DIBS

[fabxc]

This should be fixed via #3151. Unfortunately, it didn't go into beta.3 by accident. We should probably cut beta.4 for that.

[gouthamve]

Can this be closed?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.