Prometheus 2.0.0-beta.0 gaps in metrics

[hanikesn]

I'm running the prometheus 2.0.0-beta.0 in our kubernetes 1.7.1 cluster and it's having trouble with gaps in the metrics which are scraped from cadvisor. Prometheus 1 .7.0 scrapes them fine:

2.0.0-beta.0

1.7.0

Scrape rule:

- job_name: kubernetes-cadvisor
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: node
    namespaces:
      names: []

There's nothing suspicious in the logs.

[brian-brazil]

The gaps are probably real, and this is the new staleness handling. Are you sure that the containers in question were returned at every scrape?

[hanikesn]

Are you sure that the containers in question were returned at every scrape?

I need to dig into this deeper.

The gaps are probably real, and this is the new staleness handling.

You can't always control what you're scraping so I'm wondering whether it makes sense to allow overriding this behavior per target.

[brian-brazil]

I'm wondering whether it makes sense to allow overriding this behavior per target.

It's all or nothing.

[hanikesn]

The /metrics endpoint actually randomly alters between showing systemd cgroups and docker containers. So this is clearly a bug in kubernetes/cadvisor.

See:
#2916
kubernetes/kubernetes#48483

[fabxc]

The linked k8s issue does not resolve this in general I think.
Even after the fix, the cAdvisor endpoint exposes a lot of metrics only occasionally.

[dadux]

Just upgraded to 2.0.0-beta.0 seeing the same "gaps" with the cAdvisor target.

@fabxc - I'm not familiar with the staleness handling, but my understanding is in that case prometheus 1.7 metrics would be inaccurate too and therefore there's no point for us rolling back ?

[fabxc]

Yea, so cAdvisor metrics are flapping on and off. 2.0 just happens to detect this and set staleness markers.
Practically, it's just reduced sampling frequency so what 1.7 shows is not totally wrong.

Overall it's definitely a cAdvisor issue though google/cadvisor#1704

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.