Queue chucks to persist goes over max-chunks-to-persist

[dadux]

What did you do?
Configure Prometheus with max-chunks-to-persist 750000 and observe it going over 2M.

prometheus_local_storage_queued_chunks_to_persist_total 2229179
prometheus_local_storage_max_chunks_to_persist 750000

See the full /metrics on the instance.

What did you expect to see?
I was expecting rushed mode being kicked of or even throttled ingestion.

Environment
Docker Pod running in k8s. Storage on EBS volume.

• System information:
  Linux 4.9.9-coreos-r1 x86_64

• Prometheus version:
  v1.5.2

• Prometheus configuration file:
  query.max-concurrency 20
  query.staleness-delta 5m0s
  query.timeout 2m0s
  storage.local.checkpoint-dirty-series-limit 5000
  storage.local.checkpoint-interval 5m0s
  storage.local.chunk-encoding-version 1
  storage.local.dirty false
  storage.local.engine persisted
  storage.local.index-cache-size.fingerprint-to-metric 10485760
  storage.local.index-cache-size.fingerprint-to-timerange 5242880
  storage.local.index-cache-size.label-name-to-label-values 10485760
  storage.local.index-cache-size.label-pair-to-fingerprints 20971520
  storage.local.max-chunks-to-persist 750000
  storage.local.memory-chunks 1500000
  storage.local.num-fingerprint-mutexes 4096

full /metrics on the instance

# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.004999703
go_gc_duration_seconds{quantile="0.25"} 0.005487825
go_gc_duration_seconds{quantile="0.5"} 0.006005587000000001
go_gc_duration_seconds{quantile="0.75"} 0.007085386000000001
go_gc_duration_seconds{quantile="1"} 0.14327307
go_gc_duration_seconds_sum 34.104061878
go_gc_duration_seconds_count 5830
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 213
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 5.394736136e+09
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.435363500768e+13
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 3.647921e+07
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 4.1189805644e+10
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 4.92087296e+08
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 5.394736136e+09
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.257696768e+09
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 6.359949312e+09
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 2.046836e+07
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 3.07560448e+08
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.261764608e+10
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.4903135320445242e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 5.699272e+06
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 4.1210274004e+10
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 9600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 9.439664e+07
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 1.46997248e+08
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 7.299331716e+09
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 2.6442518e+07
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 3.014656e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 3.014656e+06
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.3322683392e+10
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="alertmanagers",quantile="0.5"} NaN
http_request_duration_microseconds{handler="alertmanagers",quantile="0.9"} NaN
http_request_duration_microseconds{handler="alertmanagers",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="alertmanagers"} 0
http_request_duration_microseconds_count{handler="alertmanagers"} 0
http_request_duration_microseconds{handler="alerts",quantile="0.5"} NaN
http_request_duration_microseconds{handler="alerts",quantile="0.9"} NaN
http_request_duration_microseconds{handler="alerts",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="alerts"} 6776.339
http_request_duration_microseconds_count{handler="alerts"} 2
http_request_duration_microseconds{handler="config",quantile="0.5"} NaN
http_request_duration_microseconds{handler="config",quantile="0.9"} NaN
http_request_duration_microseconds{handler="config",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="config"} 4057.0199999999995
http_request_duration_microseconds_count{handler="config"} 3
http_request_duration_microseconds{handler="consoles",quantile="0.5"} NaN
http_request_duration_microseconds{handler="consoles",quantile="0.9"} NaN
http_request_duration_microseconds{handler="consoles",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="consoles"} 0
http_request_duration_microseconds_count{handler="consoles"} 0
http_request_duration_microseconds{handler="drop_series",quantile="0.5"} NaN
http_request_duration_microseconds{handler="drop_series",quantile="0.9"} NaN
http_request_duration_microseconds{handler="drop_series",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="drop_series"} 0
http_request_duration_microseconds_count{handler="drop_series"} 0
http_request_duration_microseconds{handler="federate",quantile="0.5"} NaN
http_request_duration_microseconds{handler="federate",quantile="0.9"} NaN
http_request_duration_microseconds{handler="federate",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="federate"} 0
http_request_duration_microseconds_count{handler="federate"} 0
http_request_duration_microseconds{handler="flags",quantile="0.5"} NaN
http_request_duration_microseconds{handler="flags",quantile="0.9"} NaN
http_request_duration_microseconds{handler="flags",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="flags"} 2834.493
http_request_duration_microseconds_count{handler="flags"} 2
http_request_duration_microseconds{handler="graph",quantile="0.5"} 2494.005
http_request_duration_microseconds{handler="graph",quantile="0.9"} 2494.005
http_request_duration_microseconds{handler="graph",quantile="0.99"} 2494.005
http_request_duration_microseconds_sum{handler="graph"} 11221.242000000002
http_request_duration_microseconds_count{handler="graph"} 7
http_request_duration_microseconds{handler="heap",quantile="0.5"} NaN
http_request_duration_microseconds{handler="heap",quantile="0.9"} NaN
http_request_duration_microseconds{handler="heap",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="heap"} 0
http_request_duration_microseconds_count{handler="heap"} 0
http_request_duration_microseconds{handler="label_values",quantile="0.5"} 2247.426
http_request_duration_microseconds{handler="label_values",quantile="0.9"} 2247.426
http_request_duration_microseconds{handler="label_values",quantile="0.99"} 2247.426
http_request_duration_microseconds_sum{handler="label_values"} 734572.3990000001
http_request_duration_microseconds_count{handler="label_values"} 74
http_request_duration_microseconds{handler="options",quantile="0.5"} NaN
http_request_duration_microseconds{handler="options",quantile="0.9"} NaN
http_request_duration_microseconds{handler="options",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="options"} 0
http_request_duration_microseconds_count{handler="options"} 0
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 21213.127
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 1.732119048e+06
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 4.428101429e+06
http_request_duration_microseconds_sum{handler="prometheus"} 4.887030834034989e+09
http_request_duration_microseconds_count{handler="prometheus"} 11585
http_request_duration_microseconds{handler="query",quantile="0.5"} 1142.225
http_request_duration_microseconds{handler="query",quantile="0.9"} 3629.676
http_request_duration_microseconds{handler="query",quantile="0.99"} 3629.676
http_request_duration_microseconds_sum{handler="query"} 216227.47000000003
http_request_duration_microseconds_count{handler="query"} 16
http_request_duration_microseconds{handler="query_range",quantile="0.5"} 4292.663
http_request_duration_microseconds{handler="query_range",quantile="0.9"} 5180.249
http_request_duration_microseconds{handler="query_range",quantile="0.99"} 5180.249
http_request_duration_microseconds_sum{handler="query_range"} 1.0718708360890007e+09
http_request_duration_microseconds_count{handler="query_range"} 2242
http_request_duration_microseconds{handler="rules",quantile="0.5"} NaN
http_request_duration_microseconds{handler="rules",quantile="0.9"} NaN
http_request_duration_microseconds{handler="rules",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="rules"} 2398.597
http_request_duration_microseconds_count{handler="rules"} 1
http_request_duration_microseconds{handler="series",quantile="0.5"} NaN
http_request_duration_microseconds{handler="series",quantile="0.9"} NaN
http_request_duration_microseconds{handler="series",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="series"} 0
http_request_duration_microseconds_count{handler="series"} 0
http_request_duration_microseconds{handler="static",quantile="0.5"} 438.007
http_request_duration_microseconds{handler="static",quantile="0.9"} 438.007
http_request_duration_microseconds{handler="static",quantile="0.99"} 438.007
http_request_duration_microseconds_sum{handler="static"} 77827.608
http_request_duration_microseconds_count{handler="static"} 77
http_request_duration_microseconds{handler="status",quantile="0.5"} 1127.088
http_request_duration_microseconds{handler="status",quantile="0.9"} 2484.48
http_request_duration_microseconds{handler="status",quantile="0.99"} 5445.786
http_request_duration_microseconds_sum{handler="status"} 7.775297752400059e+07
http_request_duration_microseconds_count{handler="status"} 43754
http_request_duration_microseconds{handler="targets",quantile="0.5"} NaN
http_request_duration_microseconds{handler="targets",quantile="0.9"} NaN
http_request_duration_microseconds{handler="targets",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="targets"} 17186.902000000002
http_request_duration_microseconds_count{handler="targets"} 2
http_request_duration_microseconds{handler="version",quantile="0.5"} NaN
http_request_duration_microseconds{handler="version",quantile="0.9"} NaN
http_request_duration_microseconds{handler="version",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="version"} 0
http_request_duration_microseconds_count{handler="version"} 0
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="alertmanagers",quantile="0.5"} NaN
http_request_size_bytes{handler="alertmanagers",quantile="0.9"} NaN
http_request_size_bytes{handler="alertmanagers",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="alertmanagers"} 0
http_request_size_bytes_count{handler="alertmanagers"} 0
http_request_size_bytes{handler="alerts",quantile="0.5"} NaN
http_request_size_bytes{handler="alerts",quantile="0.9"} NaN
http_request_size_bytes{handler="alerts",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="alerts"} 802
http_request_size_bytes_count{handler="alerts"} 2
http_request_size_bytes{handler="config",quantile="0.5"} NaN
http_request_size_bytes{handler="config",quantile="0.9"} NaN
http_request_size_bytes{handler="config",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="config"} 1382
http_request_size_bytes_count{handler="config"} 3
http_request_size_bytes{handler="consoles",quantile="0.5"} NaN
http_request_size_bytes{handler="consoles",quantile="0.9"} NaN
http_request_size_bytes{handler="consoles",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="consoles"} 0
http_request_size_bytes_count{handler="consoles"} 0
http_request_size_bytes{handler="drop_series",quantile="0.5"} NaN
http_request_size_bytes{handler="drop_series",quantile="0.9"} NaN
http_request_size_bytes{handler="drop_series",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="drop_series"} 0
http_request_size_bytes_count{handler="drop_series"} 0
http_request_size_bytes{handler="federate",quantile="0.5"} NaN
http_request_size_bytes{handler="federate",quantile="0.9"} NaN
http_request_size_bytes{handler="federate",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="federate"} 0
http_request_size_bytes_count{handler="federate"} 0
http_request_size_bytes{handler="flags",quantile="0.5"} NaN
http_request_size_bytes{handler="flags",quantile="0.9"} NaN
http_request_size_bytes{handler="flags",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="flags"} 800
http_request_size_bytes_count{handler="flags"} 2
http_request_size_bytes{handler="graph",quantile="0.5"} 535
http_request_size_bytes{handler="graph",quantile="0.9"} 535
http_request_size_bytes{handler="graph",quantile="0.99"} 535
http_request_size_bytes_sum{handler="graph"} 3171
http_request_size_bytes_count{handler="graph"} 7
http_request_size_bytes{handler="heap",quantile="0.5"} NaN
http_request_size_bytes{handler="heap",quantile="0.9"} NaN
http_request_size_bytes{handler="heap",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="heap"} 0
http_request_size_bytes_count{handler="heap"} 0
http_request_size_bytes{handler="label_values",quantile="0.5"} 585
http_request_size_bytes{handler="label_values",quantile="0.9"} 585
http_request_size_bytes{handler="label_values",quantile="0.99"} 585
http_request_size_bytes_sum{handler="label_values"} 41235
http_request_size_bytes_count{handler="label_values"} 74
http_request_size_bytes{handler="options",quantile="0.5"} NaN
http_request_size_bytes{handler="options",quantile="0.9"} NaN
http_request_size_bytes{handler="options",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="options"} 0
http_request_size_bytes_count{handler="options"} 0
http_request_size_bytes{handler="prometheus",quantile="0.5"} 243
http_request_size_bytes{handler="prometheus",quantile="0.9"} 243
http_request_size_bytes{handler="prometheus",quantile="0.99"} 243
http_request_size_bytes_sum{handler="prometheus"} 2.815278e+06
http_request_size_bytes_count{handler="prometheus"} 11585
http_request_size_bytes{handler="query",quantile="0.5"} 763
http_request_size_bytes{handler="query",quantile="0.9"} 776
http_request_size_bytes{handler="query",quantile="0.99"} 776
http_request_size_bytes_sum{handler="query"} 10380
http_request_size_bytes_count{handler="query"} 16
http_request_size_bytes{handler="query_range",quantile="0.5"} 712
http_request_size_bytes{handler="query_range",quantile="0.9"} 713
http_request_size_bytes{handler="query_range",quantile="0.99"} 713
http_request_size_bytes_sum{handler="query_range"} 1.580196e+06
http_request_size_bytes_count{handler="query_range"} 2242
http_request_size_bytes{handler="rules",quantile="0.5"} NaN
http_request_size_bytes{handler="rules",quantile="0.9"} NaN
http_request_size_bytes{handler="rules",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="rules"} 620
http_request_size_bytes_count{handler="rules"} 1
http_request_size_bytes{handler="series",quantile="0.5"} NaN
http_request_size_bytes{handler="series",quantile="0.9"} NaN
http_request_size_bytes{handler="series",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="series"} 0
http_request_size_bytes_count{handler="series"} 0
http_request_size_bytes{handler="static",quantile="0.5"} 548
http_request_size_bytes{handler="static",quantile="0.9"} 548
http_request_size_bytes{handler="static",quantile="0.99"} 548
http_request_size_bytes_sum{handler="static"} 33175
http_request_size_bytes_count{handler="static"} 77
http_request_size_bytes{handler="status",quantile="0.5"} 99
http_request_size_bytes{handler="status",quantile="0.9"} 206
http_request_size_bytes{handler="status",quantile="0.99"} 206
http_request_size_bytes_sum{handler="status"} 5.293949e+06
http_request_size_bytes_count{handler="status"} 43754
http_request_size_bytes{handler="targets",quantile="0.5"} NaN
http_request_size_bytes{handler="targets",quantile="0.9"} NaN
http_request_size_bytes{handler="targets",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="targets"} 1266
http_request_size_bytes_count{handler="targets"} 2
http_request_size_bytes{handler="version",quantile="0.5"} NaN
http_request_size_bytes{handler="version",quantile="0.9"} NaN
http_request_size_bytes{handler="version",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="version"} 0
http_request_size_bytes_count{handler="version"} 0
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="alerts",method="get"} 2
http_requests_total{code="200",handler="config",method="get"} 3
http_requests_total{code="200",handler="flags",method="get"} 2
http_requests_total{code="200",handler="graph",method="get"} 7
http_requests_total{code="200",handler="label_values",method="get"} 74
http_requests_total{code="200",handler="prometheus",method="get"} 11585
http_requests_total{code="200",handler="query",method="get"} 16
http_requests_total{code="200",handler="query_range",method="get"} 2227
http_requests_total{code="200",handler="rules",method="get"} 1
http_requests_total{code="200",handler="static",method="get"} 77
http_requests_total{code="200",handler="status",method="get"} 43754
http_requests_total{code="200",handler="targets",method="get"} 2
http_requests_total{code="400",handler="query_range",method="get"} 14
http_requests_total{code="503",handler="query_range",method="get"} 1
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="alertmanagers",quantile="0.5"} NaN
http_response_size_bytes{handler="alertmanagers",quantile="0.9"} NaN
http_response_size_bytes{handler="alertmanagers",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="alertmanagers"} 0
http_response_size_bytes_count{handler="alertmanagers"} 0
http_response_size_bytes{handler="alerts",quantile="0.5"} NaN
http_response_size_bytes{handler="alerts",quantile="0.9"} NaN
http_response_size_bytes{handler="alerts",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="alerts"} 36297
http_response_size_bytes_count{handler="alerts"} 2
http_response_size_bytes{handler="config",quantile="0.5"} NaN
http_response_size_bytes{handler="config",quantile="0.9"} NaN
http_response_size_bytes{handler="config",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="config"} 32061
http_response_size_bytes_count{handler="config"} 3
http_response_size_bytes{handler="consoles",quantile="0.5"} NaN
http_response_size_bytes{handler="consoles",quantile="0.9"} NaN
http_response_size_bytes{handler="consoles",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="consoles"} 0
http_response_size_bytes_count{handler="consoles"} 0
http_response_size_bytes{handler="drop_series",quantile="0.5"} NaN
http_response_size_bytes{handler="drop_series",quantile="0.9"} NaN
http_response_size_bytes{handler="drop_series",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="drop_series"} 0
http_response_size_bytes_count{handler="drop_series"} 0
http_response_size_bytes{handler="federate",quantile="0.5"} NaN
http_response_size_bytes{handler="federate",quantile="0.9"} NaN
http_response_size_bytes{handler="federate",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="federate"} 0
http_response_size_bytes_count{handler="federate"} 0
http_response_size_bytes{handler="flags",quantile="0.5"} NaN
http_response_size_bytes{handler="flags",quantile="0.9"} NaN
http_response_size_bytes{handler="flags",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="flags"} 16336
http_response_size_bytes_count{handler="flags"} 2
http_response_size_bytes{handler="graph",quantile="0.5"} 3681
http_response_size_bytes{handler="graph",quantile="0.9"} 3681
http_response_size_bytes{handler="graph",quantile="0.99"} 3681
http_response_size_bytes_sum{handler="graph"} 25767
http_response_size_bytes_count{handler="graph"} 7
http_response_size_bytes{handler="heap",quantile="0.5"} NaN
http_response_size_bytes{handler="heap",quantile="0.9"} NaN
http_response_size_bytes{handler="heap",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="heap"} 0
http_response_size_bytes_count{handler="heap"} 0
http_response_size_bytes{handler="label_values",quantile="0.5"} 2545
http_response_size_bytes{handler="label_values",quantile="0.9"} 2545
http_response_size_bytes{handler="label_values",quantile="0.99"} 2545
http_response_size_bytes_sum{handler="label_values"} 111854
http_response_size_bytes_count{handler="label_values"} 74
http_response_size_bytes{handler="options",quantile="0.5"} NaN
http_response_size_bytes{handler="options",quantile="0.9"} NaN
http_response_size_bytes{handler="options",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="options"} 0
http_response_size_bytes_count{handler="options"} 0
http_response_size_bytes{handler="prometheus",quantile="0.5"} 6640
http_response_size_bytes{handler="prometheus",quantile="0.9"} 6663
http_response_size_bytes{handler="prometheus",quantile="0.99"} 6672
http_response_size_bytes_sum{handler="prometheus"} 7.0623133e+07
http_response_size_bytes_count{handler="prometheus"} 11585
http_response_size_bytes{handler="query",quantile="0.5"} 258
http_response_size_bytes{handler="query",quantile="0.9"} 277
http_response_size_bytes{handler="query",quantile="0.99"} 277
http_response_size_bytes_sum{handler="query"} 4313
http_response_size_bytes_count{handler="query"} 16
http_response_size_bytes{handler="query_range",quantile="0.5"} 1014
http_response_size_bytes{handler="query_range",quantile="0.9"} 2036
http_response_size_bytes{handler="query_range",quantile="0.99"} 2036
http_response_size_bytes_sum{handler="query_range"} 8.370195e+06
http_response_size_bytes_count{handler="query_range"} 2242
http_response_size_bytes{handler="rules",quantile="0.5"} NaN
http_response_size_bytes{handler="rules",quantile="0.9"} NaN
http_response_size_bytes{handler="rules",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="rules"} 19837
http_response_size_bytes_count{handler="rules"} 1
http_response_size_bytes{handler="series",quantile="0.5"} NaN
http_response_size_bytes{handler="series",quantile="0.9"} NaN
http_response_size_bytes{handler="series",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="series"} 0
http_response_size_bytes_count{handler="series"} 0
http_response_size_bytes{handler="static",quantile="0.5"} 6316
http_response_size_bytes{handler="static",quantile="0.9"} 6316
http_response_size_bytes{handler="static",quantile="0.99"} 6316
http_response_size_bytes_sum{handler="static"} 2.470569e+06
http_response_size_bytes_count{handler="static"} 77
http_response_size_bytes{handler="status",quantile="0.5"} 3863
http_response_size_bytes{handler="status",quantile="0.9"} 3863
http_response_size_bytes{handler="status",quantile="0.99"} 3863
http_response_size_bytes_sum{handler="status"} 1.69021702e+08
http_response_size_bytes_count{handler="status"} 43754
http_response_size_bytes{handler="targets",quantile="0.5"} NaN
http_response_size_bytes{handler="targets",quantile="0.9"} NaN
http_response_size_bytes{handler="targets",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="targets"} 110343
http_response_size_bytes_count{handler="targets"} 2
http_response_size_bytes{handler="version",quantile="0.5"} NaN
http_response_size_bytes{handler="version",quantile="0.9"} NaN
http_response_size_bytes{handler="version",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="version"} 0
http_response_size_bytes_count{handler="version"} 0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 82044.23
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 121
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.3031026688e+10
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.49013969068e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.3359898624e+10
# HELP prometheus_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which prometheus was built.
# TYPE prometheus_build_info gauge
prometheus_build_info{branch="master",goversion="go1.7.5",revision="bd1182d29f462c39544f94cc822830e1c64cf55b",version="1.5.2"} 1
# HELP prometheus_config_last_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE prometheus_config_last_reload_success_timestamp_seconds gauge
prometheus_config_last_reload_success_timestamp_seconds 1.490139691e+09
# HELP prometheus_config_last_reload_successful Whether the last configuration reload attempt was successful.
# TYPE prometheus_config_last_reload_successful gauge
prometheus_config_last_reload_successful 1
# HELP prometheus_engine_queries The current number of queries being executed or waiting.
# TYPE prometheus_engine_queries gauge
prometheus_engine_queries 0
# HELP prometheus_engine_queries_concurrent_max The max number of concurrent queries.
# TYPE prometheus_engine_queries_concurrent_max gauge
prometheus_engine_queries_concurrent_max 20
# HELP prometheus_evaluator_duration_seconds The duration of rule group evaluations.
# TYPE prometheus_evaluator_duration_seconds summary
prometheus_evaluator_duration_seconds{quantile="0.01"} 1.7807279930000002
prometheus_evaluator_duration_seconds{quantile="0.05"} 1.7807279930000002
prometheus_evaluator_duration_seconds{quantile="0.5"} 7.5646569580000005
prometheus_evaluator_duration_seconds{quantile="0.9"} 9.47190416
prometheus_evaluator_duration_seconds{quantile="0.99"} 10.137690307
prometheus_evaluator_duration_seconds_sum 14566.237358832037
prometheus_evaluator_duration_seconds_count 5795
# HELP prometheus_evaluator_iterations_skipped_total The total number of rule group evaluations skipped due to throttled metric storage.
# TYPE prometheus_evaluator_iterations_skipped_total counter
prometheus_evaluator_iterations_skipped_total 0
# HELP prometheus_local_storage_checkpoint_duration_seconds The duration in seconds taken for checkpointing open chunks and chunks yet to be persisted
# TYPE prometheus_local_storage_checkpoint_duration_seconds summary
prometheus_local_storage_checkpoint_duration_seconds{quantile="0.5"} 36.312239708
prometheus_local_storage_checkpoint_duration_seconds{quantile="0.9"} 36.312239708
prometheus_local_storage_checkpoint_duration_seconds{quantile="0.99"} 36.312239708
prometheus_local_storage_checkpoint_duration_seconds_sum 11916.068764412996
prometheus_local_storage_checkpoint_duration_seconds_count 541
# HELP prometheus_local_storage_checkpoint_last_duration_seconds The duration in seconds it took to last checkpoint open chunks and chunks yet to be persisted.
# TYPE prometheus_local_storage_checkpoint_last_duration_seconds gauge
prometheus_local_storage_checkpoint_last_duration_seconds 40.74061506
# HELP prometheus_local_storage_checkpoint_last_size_bytes The size of the last checkpoint of open chunks and chunks yet to be persisted
# TYPE prometheus_local_storage_checkpoint_last_size_bytes gauge
prometheus_local_storage_checkpoint_last_size_bytes 1.034420212e+09
# HELP prometheus_local_storage_checkpoint_series_chunks_written The number of chunk written per series while checkpointing open chunks and chunks yet to be persisted.
# TYPE prometheus_local_storage_checkpoint_series_chunks_written summary
prometheus_local_storage_checkpoint_series_chunks_written{quantile="0.5"} 2
prometheus_local_storage_checkpoint_series_chunks_written{quantile="0.9"} 4
prometheus_local_storage_checkpoint_series_chunks_written{quantile="0.99"} 11
prometheus_local_storage_checkpoint_series_chunks_written_sum 2.87855226e+08
prometheus_local_storage_checkpoint_series_chunks_written_count 1.5005215e+08
# HELP prometheus_local_storage_checkpointing 1 if the storage is checkpointing, 0 otherwise.
# TYPE prometheus_local_storage_checkpointing gauge
prometheus_local_storage_checkpointing 0
# HELP prometheus_local_storage_chunk_ops_total The total number of chunk operations by their type.
# TYPE prometheus_local_storage_chunk_ops_total counter
prometheus_local_storage_chunk_ops_total{type="clone"} 316097
prometheus_local_storage_chunk_ops_total{type="create"} 2.466734e+06
prometheus_local_storage_chunk_ops_total{type="evict"} 966751
prometheus_local_storage_chunk_ops_total{type="load"} 17
prometheus_local_storage_chunk_ops_total{type="persist"} 2.095748e+06
prometheus_local_storage_chunk_ops_total{type="pin"} 4.5230854e+07
prometheus_local_storage_chunk_ops_total{type="transcode"} 1.453973e+06
prometheus_local_storage_chunk_ops_total{type="unpin"} 4.5230854e+07
# HELP prometheus_local_storage_chunkdesc_ops_total The total number of chunk descriptor operations by their type.
# TYPE prometheus_local_storage_chunkdesc_ops_total counter
prometheus_local_storage_chunkdesc_ops_total{type="evict"} 324897
prometheus_local_storage_chunkdesc_ops_total{type="load"} 19
# HELP prometheus_local_storage_chunks_to_persist The current number of chunks waiting for persistence.
# TYPE prometheus_local_storage_chunks_to_persist gauge
prometheus_local_storage_chunks_to_persist 136922
# HELP prometheus_local_storage_fingerprint_mappings_total The total number of fingerprints being mapped to avoid collisions.
# TYPE prometheus_local_storage_fingerprint_mappings_total counter
prometheus_local_storage_fingerprint_mappings_total 0
# HELP prometheus_local_storage_inconsistencies_total A counter incremented each time an inconsistency in the local storage is detected. If this is greater zero, restart the server as soon as possible.
# TYPE prometheus_local_storage_inconsistencies_total counter
prometheus_local_storage_inconsistencies_total 0
# HELP prometheus_local_storage_indexing_batch_duration_seconds Quantiles for batch indexing duration in seconds.
# TYPE prometheus_local_storage_indexing_batch_duration_seconds summary
prometheus_local_storage_indexing_batch_duration_seconds{quantile="0.5"} 0.785368075
prometheus_local_storage_indexing_batch_duration_seconds{quantile="0.9"} 4.647875012
prometheus_local_storage_indexing_batch_duration_seconds{quantile="0.99"} 6.804055466
prometheus_local_storage_indexing_batch_duration_seconds_sum 12115.882880053005
prometheus_local_storage_indexing_batch_duration_seconds_count 16736
# HELP prometheus_local_storage_indexing_batch_sizes Quantiles for indexing batch sizes (number of metrics per batch).
# TYPE prometheus_local_storage_indexing_batch_sizes summary
prometheus_local_storage_indexing_batch_sizes{quantile="0.5"} 43
prometheus_local_storage_indexing_batch_sizes{quantile="0.9"} 212
prometheus_local_storage_indexing_batch_sizes{quantile="0.99"} 398
prometheus_local_storage_indexing_batch_sizes_sum 1.343416e+06
prometheus_local_storage_indexing_batch_sizes_count 16736
# HELP prometheus_local_storage_indexing_queue_capacity The capacity of the indexing queue.
# TYPE prometheus_local_storage_indexing_queue_capacity gauge
prometheus_local_storage_indexing_queue_capacity 262144
# HELP prometheus_local_storage_indexing_queue_length The number of metrics waiting to be indexed.
# TYPE prometheus_local_storage_indexing_queue_length gauge
prometheus_local_storage_indexing_queue_length 0
# HELP prometheus_local_storage_ingested_samples_total The total number of samples ingested.
# TYPE prometheus_local_storage_ingested_samples_total counter
prometheus_local_storage_ingested_samples_total 5.1525773e+08
# HELP prometheus_local_storage_maintain_series_duration_seconds The duration in seconds it took to perform maintenance on a series.
# TYPE prometheus_local_storage_maintain_series_duration_seconds summary
prometheus_local_storage_maintain_series_duration_seconds{location="archived",quantile="0.5"} NaN
prometheus_local_storage_maintain_series_duration_seconds{location="archived",quantile="0.9"} NaN
prometheus_local_storage_maintain_series_duration_seconds{location="archived",quantile="0.99"} NaN
prometheus_local_storage_maintain_series_duration_seconds_sum{location="archived"} 0
prometheus_local_storage_maintain_series_duration_seconds_count{location="archived"} 0
prometheus_local_storage_maintain_series_duration_seconds{location="memory",quantile="0.5"} 0.00017956300000000002
prometheus_local_storage_maintain_series_duration_seconds{location="memory",quantile="0.9"} 0.004196938
prometheus_local_storage_maintain_series_duration_seconds{location="memory",quantile="0.99"} 0.024444243
prometheus_local_storage_maintain_series_duration_seconds_sum{location="memory"} 11457.260316394357
prometheus_local_storage_maintain_series_duration_seconds_count{location="memory"} 5.089391e+06
# HELP prometheus_local_storage_max_chunks_to_persist The maximum number of chunks that can be waiting for persistence before sample ingestion will stop.
# TYPE prometheus_local_storage_max_chunks_to_persist gauge
prometheus_local_storage_max_chunks_to_persist 750000
# HELP prometheus_local_storage_max_memory_chunks The configured maximum number of chunks that can be held in memory
# TYPE prometheus_local_storage_max_memory_chunks gauge
prometheus_local_storage_max_memory_chunks 1.5e+06
# HELP prometheus_local_storage_memory_chunkdescs The current number of chunk descriptors in memory.
# TYPE prometheus_local_storage_memory_chunkdescs gauge
prometheus_local_storage_memory_chunkdescs 2.141856e+06
# HELP prometheus_local_storage_memory_chunks The current number of chunks in memory, excluding cloned chunks (i.e. chunks without a descriptor).
# TYPE prometheus_local_storage_memory_chunks gauge
prometheus_local_storage_memory_chunks 1.5e+06
# HELP prometheus_local_storage_memory_dirty_series The current number of series that would require a disk seek during crash recovery.
# TYPE prometheus_local_storage_memory_dirty_series gauge
prometheus_local_storage_memory_dirty_series 1026
# HELP prometheus_local_storage_memory_series The current number of series in memory.
# TYPE prometheus_local_storage_memory_series gauge
prometheus_local_storage_memory_series 961986
# HELP prometheus_local_storage_non_existent_series_matches_total How often a non-existent series was referred to during label matching or chunk preloading. This is an indication of outdated label indexes.
# TYPE prometheus_local_storage_non_existent_series_matches_total counter
prometheus_local_storage_non_existent_series_matches_total 0
# HELP prometheus_local_storage_out_of_order_samples_total The total number of samples that were discarded because their timestamps were at or before the last received sample for a series.
# TYPE prometheus_local_storage_out_of_order_samples_total counter
prometheus_local_storage_out_of_order_samples_total{reason="multiple_values_for_timestamp"} 0
prometheus_local_storage_out_of_order_samples_total{reason="timestamp_out_of_order"} 0
# HELP prometheus_local_storage_persist_errors_total The total number of errors while persisting chunks.
# TYPE prometheus_local_storage_persist_errors_total counter
prometheus_local_storage_persist_errors_total 0
# HELP prometheus_local_storage_persistence_urgency_score A score of urgency to persist chunks, 0 is least urgent, 1 most.
# TYPE prometheus_local_storage_persistence_urgency_score gauge
prometheus_local_storage_persistence_urgency_score 0.182564
# HELP prometheus_local_storage_queued_chunks_to_persist_total The total number of chunks queued for persistence.
# TYPE prometheus_local_storage_queued_chunks_to_persist_total counter
prometheus_local_storage_queued_chunks_to_persist_total 2.232671e+06
# HELP prometheus_local_storage_rushed_mode 1 if the storage is in rushed mode, 0 otherwise. In rushed mode, the system behaves as if the persistence_urgency_score is 1.
# TYPE prometheus_local_storage_rushed_mode gauge
prometheus_local_storage_rushed_mode 0
# HELP prometheus_local_storage_series_chunks_persisted The number of chunks persisted per series.
# TYPE prometheus_local_storage_series_chunks_persisted histogram
prometheus_local_storage_series_chunks_persisted_bucket{le="1"} 1.406696e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="2"} 1.569977e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="4"} 1.62136e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="8"} 1.643413e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="16"} 1.649224e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="32"} 1.649224e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="64"} 1.649224e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="128"} 1.649224e+06
prometheus_local_storage_series_chunks_persisted_bucket{le="+Inf"} 1.649224e+06
prometheus_local_storage_series_chunks_persisted_sum 2.095748e+06
prometheus_local_storage_series_chunks_persisted_count 1.649224e+06
# HELP prometheus_local_storage_series_ops_total The total number of series operations by their type.
# TYPE prometheus_local_storage_series_ops_total counter
prometheus_local_storage_series_ops_total{type="archive"} 381444
prometheus_local_storage_series_ops_total{type="create"} 1.343416e+06
prometheus_local_storage_series_ops_total{type="maintenance_in_archive"} 0
prometheus_local_storage_series_ops_total{type="maintenance_in_memory"} 5.089391e+06
prometheus_local_storage_series_ops_total{type="purge_from_archive"} 0
prometheus_local_storage_series_ops_total{type="purge_from_memory"} 0
prometheus_local_storage_series_ops_total{type="purge_on_request"} 0
prometheus_local_storage_series_ops_total{type="quarantine_completed"} 0
prometheus_local_storage_series_ops_total{type="quarantine_dropped"} 0
prometheus_local_storage_series_ops_total{type="quarantine_failed"} 0
prometheus_local_storage_series_ops_total{type="unarchive"} 14
# HELP prometheus_local_storage_started_dirty Whether the local storage was found to be dirty (and crash recovery occurred) during Prometheus startup.
# TYPE prometheus_local_storage_started_dirty gauge
prometheus_local_storage_started_dirty 0
# HELP prometheus_notifications_dropped_total Total number of alerts dropped due to errors when sending to Alertmanager.
# TYPE prometheus_notifications_dropped_total counter
prometheus_notifications_dropped_total 0
# HELP prometheus_notifications_latency_seconds Latency quantiles for sending alert notifications (not including dropped notifications).
# TYPE prometheus_notifications_latency_seconds summary
prometheus_notifications_latency_seconds{alertmanager="http://alertmanager:9093/api/v1/alerts",quantile="0.5"} 0.036391399000000005
prometheus_notifications_latency_seconds{alertmanager="http://alertmanager:9093/api/v1/alerts",quantile="0.9"} 0.104539406
prometheus_notifications_latency_seconds{alertmanager="http://alertmanager:9093/api/v1/alerts",quantile="0.99"} 0.181000094
prometheus_notifications_latency_seconds_sum{alertmanager="http://alertmanager:9093/api/v1/alerts"} 222.7537781500002
prometheus_notifications_latency_seconds_count{alertmanager="http://alertmanager:9093/api/v1/alerts"} 1954
# HELP prometheus_notifications_queue_capacity The capacity of the alert notifications queue.
# TYPE prometheus_notifications_queue_capacity gauge
prometheus_notifications_queue_capacity 10000
# HELP prometheus_notifications_queue_length The number of alert notifications in the queue.
# TYPE prometheus_notifications_queue_length gauge
prometheus_notifications_queue_length 0
# HELP prometheus_notifications_sent_total Total number of alerts successfully sent.
# TYPE prometheus_notifications_sent_total counter
prometheus_notifications_sent_total{alertmanager="http://alertmanager:9093/api/v1/alerts"} 2917
# HELP prometheus_rule_evaluation_duration_seconds The duration for a rule to execute.
# TYPE prometheus_rule_evaluation_duration_seconds summary
prometheus_rule_evaluation_duration_seconds{rule_type="alerting",quantile="0.5"} 0.00044825100000000004
prometheus_rule_evaluation_duration_seconds{rule_type="alerting",quantile="0.9"} 0.022008334
prometheus_rule_evaluation_duration_seconds{rule_type="alerting",quantile="0.99"} 0.11479209400000001
prometheus_rule_evaluation_duration_seconds_sum{rule_type="alerting"} 2850.3644551380107
prometheus_rule_evaluation_duration_seconds_count{rule_type="alerting"} 81130
prometheus_rule_evaluation_duration_seconds{rule_type="recording",quantile="0.5"} 2.492824734
prometheus_rule_evaluation_duration_seconds{rule_type="recording",quantile="0.9"} 8.1046989
prometheus_rule_evaluation_duration_seconds{rule_type="recording",quantile="0.99"} 9.728057687
prometheus_rule_evaluation_duration_seconds_sum{rule_type="recording"} 115385.48707357487
prometheus_rule_evaluation_duration_seconds_count{rule_type="recording"} 86925
# HELP prometheus_rule_evaluation_failures_total The total number of rule evaluation failures.
# TYPE prometheus_rule_evaluation_failures_total counter
prometheus_rule_evaluation_failures_total{rule_type="alerting"} 0
prometheus_rule_evaluation_failures_total{rule_type="recording"} 0
# HELP prometheus_sd_azure_refresh_duration_seconds The duration of a Azure-SD refresh in seconds.
# TYPE prometheus_sd_azure_refresh_duration_seconds summary
prometheus_sd_azure_refresh_duration_seconds{quantile="0.5"} NaN
prometheus_sd_azure_refresh_duration_seconds{quantile="0.9"} NaN
prometheus_sd_azure_refresh_duration_seconds{quantile="0.99"} NaN
prometheus_sd_azure_refresh_duration_seconds_sum 0
prometheus_sd_azure_refresh_duration_seconds_count 0
# HELP prometheus_sd_azure_refresh_failures_total Number of Azure-SD refresh failures.
# TYPE prometheus_sd_azure_refresh_failures_total counter
prometheus_sd_azure_refresh_failures_total 0
# HELP prometheus_sd_consul_rpc_duration_seconds The duration of a Consul RPC call in seconds.
# TYPE prometheus_sd_consul_rpc_duration_seconds summary
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.5"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.9"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.99"} NaN
prometheus_sd_consul_rpc_duration_seconds_sum{call="service",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds_count{call="service",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.5"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.9"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.99"} NaN
prometheus_sd_consul_rpc_duration_seconds_sum{call="services",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds_count{call="services",endpoint="catalog"} 0
# HELP prometheus_sd_consul_rpc_failures_total The number of Consul RPC call failures.
# TYPE prometheus_sd_consul_rpc_failures_total counter
prometheus_sd_consul_rpc_failures_total 0
# HELP prometheus_sd_dns_lookup_failures_total The number of DNS-SD lookup failures.
# TYPE prometheus_sd_dns_lookup_failures_total counter
prometheus_sd_dns_lookup_failures_total 0
# HELP prometheus_sd_dns_lookups_total The number of DNS-SD lookups.
# TYPE prometheus_sd_dns_lookups_total counter
prometheus_sd_dns_lookups_total 0
# HELP prometheus_sd_ec2_refresh_duration_seconds The duration of a EC2-SD refresh in seconds.
# TYPE prometheus_sd_ec2_refresh_duration_seconds summary
prometheus_sd_ec2_refresh_duration_seconds{quantile="0.5"} NaN
prometheus_sd_ec2_refresh_duration_seconds{quantile="0.9"} NaN
prometheus_sd_ec2_refresh_duration_seconds{quantile="0.99"} NaN
prometheus_sd_ec2_refresh_duration_seconds_sum 0
prometheus_sd_ec2_refresh_duration_seconds_count 0
# HELP prometheus_sd_ec2_refresh_failures_total The number of EC2-SD scrape failures.
# TYPE prometheus_sd_ec2_refresh_failures_total counter
prometheus_sd_ec2_refresh_failures_total 0
# HELP prometheus_sd_file_read_errors_total The number of File-SD read errors.
# TYPE prometheus_sd_file_read_errors_total counter
prometheus_sd_file_read_errors_total 0
# HELP prometheus_sd_file_scan_duration_seconds The duration of the File-SD scan in seconds.
# TYPE prometheus_sd_file_scan_duration_seconds summary
prometheus_sd_file_scan_duration_seconds{quantile="0.5"} NaN
prometheus_sd_file_scan_duration_seconds{quantile="0.9"} NaN
prometheus_sd_file_scan_duration_seconds{quantile="0.99"} NaN
prometheus_sd_file_scan_duration_seconds_sum 0
prometheus_sd_file_scan_duration_seconds_count 0
# HELP prometheus_sd_gce_refresh_duration The duration of a GCE-SD refresh in seconds.
# TYPE prometheus_sd_gce_refresh_duration summary
prometheus_sd_gce_refresh_duration{quantile="0.5"} NaN
prometheus_sd_gce_refresh_duration{quantile="0.9"} NaN
prometheus_sd_gce_refresh_duration{quantile="0.99"} NaN
prometheus_sd_gce_refresh_duration_sum 0
prometheus_sd_gce_refresh_duration_count 0
# HELP prometheus_sd_gce_refresh_failures_total The number of GCE-SD refresh failures.
# TYPE prometheus_sd_gce_refresh_failures_total counter
prometheus_sd_gce_refresh_failures_total 0
# HELP prometheus_sd_kubernetes_events_total The number of Kubernetes events handled.
# TYPE prometheus_sd_kubernetes_events_total counter
prometheus_sd_kubernetes_events_total{event="add",role="endpoints"} 117
prometheus_sd_kubernetes_events_total{event="add",role="node"} 5
prometheus_sd_kubernetes_events_total{event="add",role="pod"} 37006
prometheus_sd_kubernetes_events_total{event="add",role="service"} 148
prometheus_sd_kubernetes_events_total{event="delete",role="endpoints"} 21
prometheus_sd_kubernetes_events_total{event="delete",role="node"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="pod"} 30848
prometheus_sd_kubernetes_events_total{event="delete",role="service"} 28
prometheus_sd_kubernetes_events_total{event="update",role="endpoints"} 550464
prometheus_sd_kubernetes_events_total{event="update",role="node"} 88165
prometheus_sd_kubernetes_events_total{event="update",role="pod"} 1.936616e+06
prometheus_sd_kubernetes_events_total{event="update",role="service"} 39160
# HELP prometheus_sd_marathon_refresh_duration_seconds The duration of a Marathon-SD refresh in seconds.
# TYPE prometheus_sd_marathon_refresh_duration_seconds summary
prometheus_sd_marathon_refresh_duration_seconds{quantile="0.5"} NaN
prometheus_sd_marathon_refresh_duration_seconds{quantile="0.9"} NaN
prometheus_sd_marathon_refresh_duration_seconds{quantile="0.99"} NaN
prometheus_sd_marathon_refresh_duration_seconds_sum 0
prometheus_sd_marathon_refresh_duration_seconds_count 0
# HELP prometheus_sd_marathon_refresh_failures_total The number of Marathon-SD refresh failures.
# TYPE prometheus_sd_marathon_refresh_failures_total counter
prometheus_sd_marathon_refresh_failures_total 0
# HELP prometheus_sd_triton_refresh_duration_seconds The duration of a Triton-SD refresh in seconds.
# TYPE prometheus_sd_triton_refresh_duration_seconds summary
prometheus_sd_triton_refresh_duration_seconds{quantile="0.5"} NaN
prometheus_sd_triton_refresh_duration_seconds{quantile="0.9"} NaN
prometheus_sd_triton_refresh_duration_seconds{quantile="0.99"} NaN
prometheus_sd_triton_refresh_duration_seconds_sum 0
prometheus_sd_triton_refresh_duration_seconds_count 0
# HELP prometheus_sd_triton_refresh_failures_total The number of Triton-SD scrape failures.
# TYPE prometheus_sd_triton_refresh_failures_total counter
prometheus_sd_triton_refresh_failures_total 0
# HELP prometheus_target_interval_length_seconds Actual intervals between scrapes.
# TYPE prometheus_target_interval_length_seconds summary
prometheus_target_interval_length_seconds{interval="30s",quantile="0.01"} 29.832811516
prometheus_target_interval_length_seconds{interval="30s",quantile="0.05"} 29.911925244
prometheus_target_interval_length_seconds{interval="30s",quantile="0.5"} 30.00001697
prometheus_target_interval_length_seconds{interval="30s",quantile="0.9"} 30.063639169
prometheus_target_interval_length_seconds{interval="30s",quantile="0.99"} 30.100650676
prometheus_target_interval_length_seconds_sum{interval="30s"} 2.9550039235430467e+06
prometheus_target_interval_length_seconds_count{interval="30s"} 98500
# HELP prometheus_target_scrape_pool_sync_total Total number of syncs that were executed on a scrape pool.
# TYPE prometheus_target_scrape_pool_sync_total counter
prometheus_target_scrape_pool_sync_total{scrape_job="etcd-k8s"} 1
prometheus_target_scrape_pool_sync_total{scrape_job="kube-dns"} 701
prometheus_target_scrape_pool_sync_total{scrape_job="kubelet"} 34721
prometheus_target_scrape_pool_sync_total{scrape_job="kubernetes-apiservers"} 701
prometheus_target_scrape_pool_sync_total{scrape_job="kubernetes-pods"} 30023
prometheus_target_scrape_pool_sync_total{scrape_job="kubernetes-service-endpoints"} 701
prometheus_target_scrape_pool_sync_total{scrape_job="kubernetes-services"} 582
prometheus_target_scrape_pool_sync_total{scrape_job="prometheus"} 29994
# HELP prometheus_target_scrapes_exceeded_sample_limit_total Total number of scrapes that hit the sample limit and were rejected.
# TYPE prometheus_target_scrapes_exceeded_sample_limit_total counter
prometheus_target_scrapes_exceeded_sample_limit_total 0
# HELP prometheus_target_skipped_scrapes_total Total number of scrapes that were skipped because the metric storage was throttled.
# TYPE prometheus_target_skipped_scrapes_total counter
prometheus_target_skipped_scrapes_total 0
# HELP prometheus_target_sync_length_seconds Actual interval to sync the scrape pool.
# TYPE prometheus_target_sync_length_seconds summary
prometheus_target_sync_length_seconds{scrape_job="etcd-k8s",quantile="0.01"} NaN
prometheus_target_sync_length_seconds{scrape_job="etcd-k8s",quantile="0.05"} NaN
prometheus_target_sync_length_seconds{scrape_job="etcd-k8s",quantile="0.5"} NaN
prometheus_target_sync_length_seconds{scrape_job="etcd-k8s",quantile="0.9"} NaN
prometheus_target_sync_length_seconds{scrape_job="etcd-k8s",quantile="0.99"} NaN
prometheus_target_sync_length_seconds_sum{scrape_job="etcd-k8s"} 7.5158e-05
prometheus_target_sync_length_seconds_count{scrape_job="etcd-k8s"} 1
prometheus_target_sync_length_seconds{scrape_job="kube-dns",quantile="0.01"} 0.001412269
prometheus_target_sync_length_seconds{scrape_job="kube-dns",quantile="0.05"} 0.001412269
prometheus_target_sync_length_seconds{scrape_job="kube-dns",quantile="0.5"} 0.001457966
prometheus_target_sync_length_seconds{scrape_job="kube-dns",quantile="0.9"} 0.001554154
prometheus_target_sync_length_seconds{scrape_job="kube-dns",quantile="0.99"} 0.001554154
prometheus_target_sync_length_seconds_sum{scrape_job="kube-dns"} 2.7353538769999988
prometheus_target_sync_length_seconds_count{scrape_job="kube-dns"} 701
prometheus_target_sync_length_seconds{scrape_job="kubelet",quantile="0.01"} 0.00024942700000000004
prometheus_target_sync_length_seconds{scrape_job="kubelet",quantile="0.05"} 0.000253635
prometheus_target_sync_length_seconds{scrape_job="kubelet",quantile="0.5"} 0.00028658700000000004
prometheus_target_sync_length_seconds{scrape_job="kubelet",quantile="0.9"} 0.0007319620000000001
prometheus_target_sync_length_seconds{scrape_job="kubelet",quantile="0.99"} 0.000867736
prometheus_target_sync_length_seconds_sum{scrape_job="kubelet"} 14.01208462199998
prometheus_target_sync_length_seconds_count{scrape_job="kubelet"} 34721
prometheus_target_sync_length_seconds{scrape_job="kubernetes-apiservers",quantile="0.01"} 0.0009387890000000001
prometheus_target_sync_length_seconds{scrape_job="kubernetes-apiservers",quantile="0.05"} 0.0009387890000000001
prometheus_target_sync_length_seconds{scrape_job="kubernetes-apiservers",quantile="0.5"} 0.000960215
prometheus_target_sync_length_seconds{scrape_job="kubernetes-apiservers",quantile="0.9"} 0.001348447
prometheus_target_sync_length_seconds{scrape_job="kubernetes-apiservers",quantile="0.99"} 0.001348447
prometheus_target_sync_length_seconds_sum{scrape_job="kubernetes-apiservers"} 1.747431056
prometheus_target_sync_length_seconds_count{scrape_job="kubernetes-apiservers"} 701
prometheus_target_sync_length_seconds{scrape_job="kubernetes-pods",quantile="0.01"} 0.023539191
prometheus_target_sync_length_seconds{scrape_job="kubernetes-pods",quantile="0.05"} 0.024270679
prometheus_target_sync_length_seconds{scrape_job="kubernetes-pods",quantile="0.5"} 0.029051069000000002
prometheus_target_sync_length_seconds{scrape_job="kubernetes-pods",quantile="0.9"} 0.38018768900000005
prometheus_target_sync_length_seconds{scrape_job="kubernetes-pods",quantile="0.99"} 0.6060105210000001
prometheus_target_sync_length_seconds_sum{scrape_job="kubernetes-pods"} 2340.840363559009
prometheus_target_sync_length_seconds_count{scrape_job="kubernetes-pods"} 30023
prometheus_target_sync_length_seconds{scrape_job="kubernetes-service-endpoints",quantile="0.01"} 0.0009438030000000001
prometheus_target_sync_length_seconds{scrape_job="kubernetes-service-endpoints",quantile="0.05"} 0.0009438030000000001
prometheus_target_sync_length_seconds{scrape_job="kubernetes-service-endpoints",quantile="0.5"} 0.000986997
prometheus_target_sync_length_seconds{scrape_job="kubernetes-service-endpoints",quantile="0.9"} 0.001328111
prometheus_target_sync_length_seconds{scrape_job="kubernetes-service-endpoints",quantile="0.99"} 0.001328111
prometheus_target_sync_length_seconds_sum{scrape_job="kubernetes-service-endpoints"} 1.455830366000002
prometheus_target_sync_length_seconds_count{scrape_job="kubernetes-service-endpoints"} 701
prometheus_target_sync_length_seconds{scrape_job="kubernetes-services",quantile="0.01"} 0.000246581
prometheus_target_sync_length_seconds{scrape_job="kubernetes-services",quantile="0.05"} 0.000246581
prometheus_target_sync_length_seconds{scrape_job="kubernetes-services",quantile="0.5"} 0.000246581
prometheus_target_sync_length_seconds{scrape_job="kubernetes-services",quantile="0.9"} 0.000246581
prometheus_target_sync_length_seconds{scrape_job="kubernetes-services",quantile="0.99"} 0.000246581
prometheus_target_sync_length_seconds_sum{scrape_job="kubernetes-services"} 0.20318335299999984
prometheus_target_sync_length_seconds_count{scrape_job="kubernetes-services"} 582
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.01"} 0.02519487
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.05"} 0.025989449
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.5"} 0.029218693
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.9"} 0.398475419
prometheus_target_sync_length_seconds{scrape_job="prometheus",quantile="0.99"} 0.7670197120000001
prometheus_target_sync_length_seconds_sum{scrape_job="prometheus"} 2492.4482201490036
prometheus_target_sync_length_seconds_count{scrape_job="prometheus"} 29994
# HELP prometheus_treecache_watcher_goroutines The current number of watcher goroutines.
# TYPE prometheus_treecache_watcher_goroutines gauge
prometheus_treecache_watcher_goroutines 0
# HELP prometheus_treecache_zookeeper_failures_total The total number of ZooKeeper failures.
# TYPE prometheus_treecache_zookeeper_failures_total counter
prometheus_treecache_zookeeper_failures_total 0```

[juliusv]

I'm confused. You mention a metric prometheus_local_storage_queued_chunks_to_persist_total 2229179, but Prometheus has no metric by that name (it is also not listed in your /metrics output - EDIT: wrong, I didn't look correctly). The current number of chunks to persist is prometheus_local_storage_chunks_to_persist, which only has a value of 136922 in your /metrics example. This would be fine if your max is 750000.

[juliusv]

Oh. It seems the prometheus_local_storage_queued_chunks_to_persist_total indeed exists now, it got added recently. But it's not the one you want. That one is a total counter over the process lifetime. prometheus_local_storage_chunks_to_persist is the correct one.

[juliusv]

Closing as this seems to have been a misunderstanding about the right metric to look at.

[dadux]

wow, thanks @juliusv for the speedy answer ! Sorry for the misunderstanding.

[juliusv]

No worries!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.