Prometheus does not respect global scrape_interval?

[krogon-dp]

What did you do?

1. Set scrape_interval to 1m in global settings.
2. Query single metric for last 10 minutes. Metric is coming from cadvisor (kubelet build-in)
   • used prometheus-query for CLI

What did you expect to see?
Single metric for each minute of scrape

What did you see instead? Under which circumstances?
The scrape seems to gather metrics each 15s

Environment

Prometheus running on kubernetes 1.7.0 from helm chart (version stable/4.5.0).

• System information:

  Linux 4.4.65-k8s x86_64

• Prometheus version:

# prometheus -version
prometheus, version 1.7.1 (branch: master, revision: 3afb3fffa3a29c3de865e1172fb740442e9d0133)
  build user:       root@0aa1b7fc430d
  build date:       20170612-11:44:05
  go version:       go1.8.3

• Alertmanager version:

# alertmanager -version
alertmanager, version 0.8.0 (branch: HEAD, revision: 74e7e48d24bddd2e2a80c7840af9b2de271cc74c)
  build user:       root@439065dc2905
  build date:       20170720-14:14:06
  go version:       go1.8.3

• Prometheus configuration file (generated on the "Configuration" page

global:
  scrape_interval: 1m
  scrape_timeout: 30s
  evaluation_interval: 1m
[...]
scrape_configs:
- job_name: kubernetes-cadvisor
  scrape_interval: 1m
  scrape_timeout: 30s
  metrics_path: /metrics
  scheme: https
  kubernetes_sd_configs:
  - api_server: null
    role: node
    namespaces:
      names: []
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: false
  relabel_configs:
  - source_labels: []
    separator: ;
    regex: __meta_kubernetes_node_label_(.+)
    replacement: $1
    action: labelmap
  - source_labels: []
    separator: ;
    regex: (.*)
    target_label: __address__
    replacement: kubernetes.default.svc:443
    action: replace
  - source_labels: [__meta_kubernetes_node_name]
    separator: ;
    regex: (.+)
    target_label: __metrics_path__
    replacement: /api/v1/nodes/${1}:4194/proxy/metrics
    action: replace

• Additional information

# QUERY="container_cpu_usage_seconds_total{container_name='prometheus-server',cpu='cpu00'}"
# ./prometheus-query -query "$QUERY" -start "10 minute ago" -format csv | awk -F"," 'BEGIN {OFS=","} {if(NR==1){print "DATE_DELTA,DATE,VALUE"} if(NR>1){ if(ts==0){ts=$1} diff=$1-ts;ts=$1; $1=strftime("%Y-%m-%d %H:%M:%S", $1); last=$2; print diff, $1, $2}}' | head
DATE_DELTA,DATE,VALUE
0,2017-09-20 06:39:18,85268.771556
15,2017-09-20 06:39:33,85268.771556
15,2017-09-20 06:39:48,85268.771556
15,2017-09-20 06:40:03,85268.771556
15,2017-09-20 06:40:18,85268.771556
15,2017-09-20 06:40:33,85268.771556
15,2017-09-20 06:40:48,85268.771556
15,2017-09-20 06:41:03,85268.771556
15,2017-09-20 06:41:18,85268.771556

The same happens when rate() function is applied

# QUERY="rate(container_cpu_usage_seconds_total{container_name='prometheus-server',cpu='cpu00'}[2m])"
# ./prometheus-query -query "$QUERY" -start "10 minute ago" -format csv | awk -F"," 'BEGIN {OFS=","} {if(NR==1){print "DATE_DELTA,DATE,VALUE"} if(NR>1){ if(ts==0){ts=$1} diff=$1-ts;ts=$1; $1=strftime("%Y-%m-%d %H:%M:%S", $1); last=$2; print diff, $1, $2}}' | head
DATE_DELTA,DATE,VALUE
0,2017-09-20 06:50:23,0.063770
15,2017-09-20 06:50:38,0.059087
15,2017-09-20 06:50:53,0.054404
15,2017-09-20 06:51:08,0.049722
15,2017-09-20 06:51:23,0.036974
15,2017-09-20 06:51:38,0.033130
15,2017-09-20 06:51:53,0.029286
15,2017-09-20 06:52:08,0.025441
15,2017-09-20 06:52:23,0.073331

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.