prometheus won't restart

[korovkin]

look like our prometheus process doesn't want to restart (su sv restart prometheus),
the last time this happened we killed the process and lost a few days worth of metrics to "orphaned data".

please advice ?

1. how to properly restart to minimize the chances of the above?
2. how to debug this?

sudo sv restart prometheus
timeout: run: prometheus: (pid 18068) 667334s, got TERM

tail -F /var/log/prometheus/current
time="2017-09-19T00:53:19-07:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-09-19T00:53:19-07:00" level=info msg="Done checkpointing in-memory metrics and chunks in 510.721933ms." source="persistence.go:665"
time="2017-09-19T00:58:19-07:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-09-19T00:58:20-07:00" level=info msg="Done checkpointing in-memory metrics and chunks in 512.158393ms." source="persistence.go:665"
time="2017-09-19T01:03:20-07:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-09-19T01:03:21-07:00" level=info msg="Done checkpointing in-memory metrics and chunks in 512.980993ms." source="persistence.go:665"
time="2017-09-19T01:08:21-07:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-09-19T01:08:21-07:00" level=info msg="Done checkpointing in-memory metrics and chunks in 515.094174ms." source="persistence.go:665"
time="2017-09-19T01:13:21-07:00" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
time="2017-09-19T01:13:22-07:00" level=info msg="Done checkpointing in-memory metrics and chunks in 540.679888ms." source="persistence.go:665"

[korovkin]

Uptime | 2017-09-16 00:13:00.154769147 +0000 UTC
-- | --
/home/ubuntu



Version | 1.7.1
-- | --
3afb3fffa3a29c3de865e1172fb740442e9d0133
master
root@0aa1b7fc430d
20170612-11:44:05
go1.8.3

[korovkin]

ps aux  | grep prometheus
root      1050  0.0  0.0   4396  1108 ?        Ss   Sep06   0:17 runsvdir -P /etc/service log: urrent: /var/log/prometheus: permission denied svlogd: pausing: unable to set mode of current: /var/log/prometheus: permission denied svlogd: pausing: unable to set mode of current: /var/log/prometheus: permission denied svlogd: pausing: unable to set mode of current: /var/log/prometheus: permission denied svlogd: pausing: unable to set mode of current: /var/log/prometheus: permission denied
root      1072  0.0  0.0   4244   700 ?        Ss   Sep06   0:00 runsv prometheus
ubuntu    1081  0.0  0.0   4388   708 ?        S    Sep06   0:04 svlogd /var/log/prometheus
prometh+  1191  0.0  0.0  18852   200 ?        S    Sep06   0:00 /usr/bin/daemon --name=prometheus-pushgateway --output=/var/log/prometheus/prometheus-pushgateway.log --pidfile=/var/run/prometheus/prometheus-pushgateway.pid --user=prometheus -- /usr/bin/prometheus-pushgateway
prometh+  1192 10.1  0.1 307884 21852 ?        Sl   Sep06 2443:24 /usr/bin/prometheus-pushgateway
prometh+  1194  0.0  0.0  18852   200 ?        S    Sep06   0:00 /usr/bin/daemon --name=prometheus-node-exporter --output=/var/log/prometheus/prometheus-node-exporter.log --pidfile=/var/run/prometheus/prometheus-node-exporter.pid --user=prometheus -- /usr/bin/prometheus-node-exporter -collector.diskstats.ignored-devices="^(ram|loop|fd)\\d+$
prometh+  1195  0.0  0.1 379976 22232 ?        Sl   Sep06  14:28 /usr/bin/prometheus-node-exporter -collector.diskstats.ignored-devices="^(ram|loop|fd)\\d+$
ubuntu    6331  0.0  0.0  12944  1016 pts/1    R+   10:38   0:00 grep --color=auto prometheus
ubuntu   18068  2.8 13.6 2351548 2245984 ?     Sl   Sep15 315:23 /home/ubuntu/prometheus-1.7.1.linux-amd64/prometheus -config.file=/home/ubuntu/prometheus-1.7.1.linux-amd64/prometheus.yml -storage.local.path=/home/ubuntu/prometheus_metrics -storage.local.retention=720h --alertmanager.url http://localhost:9093 --storage.local.dirty

[korovkin]

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.1641e-05
go_gc_duration_seconds{quantile="0.25"} 9.0878e-05
go_gc_duration_seconds{quantile="0.5"} 0.00012176
go_gc_duration_seconds{quantile="0.75"} 0.000247694
go_gc_duration_seconds{quantile="1"} 0.006662402
go_gc_duration_seconds_sum 1.5321950800000002
go_gc_duration_seconds_count 7756
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 169
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.492186592e+09
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.76491935388e+12
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 3.299168e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 3.3530770048e+10
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0007362494453348106
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 1.18042624e+08
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.492186592e+09
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.7755264e+08
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 1.7229824e+09
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 5.284207e+06
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 3.74800384e+08
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 2.20053504e+09
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.5061896998667607e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 2.715496e+06
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 3.3536054255e+10
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 4800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 3.3425864e+07
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 3.9944192e+07
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 2.067545961e+09
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 6.83292e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 1.933312e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 1.933312e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.37060364e+09
# HELP go_threads Number of OS threads created
# TYPE go_threads gauge
go_threads 13
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="alertmanagers",quantile="0.5"} NaN
http_request_duration_microseconds{handler="alertmanagers",quantile="0.9"} NaN
http_request_duration_microseconds{handler="alertmanagers",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="alertmanagers"} 0
http_request_duration_microseconds_count{handler="alertmanagers"} 0
http_request_duration_microseconds{handler="alerts",quantile="0.5"} NaN
http_request_duration_microseconds{handler="alerts",quantile="0.9"} NaN
http_request_duration_microseconds{handler="alerts",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="alerts"} 1237.233
http_request_duration_microseconds_count{handler="alerts"} 1
http_request_duration_microseconds{handler="config",quantile="0.5"} NaN
http_request_duration_microseconds{handler="config",quantile="0.9"} NaN
http_request_duration_microseconds{handler="config",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="config"} 0
http_request_duration_microseconds_count{handler="config"} 0
http_request_duration_microseconds{handler="consoles",quantile="0.5"} NaN
http_request_duration_microseconds{handler="consoles",quantile="0.9"} NaN
http_request_duration_microseconds{handler="consoles",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="consoles"} 0
http_request_duration_microseconds_count{handler="consoles"} 0
http_request_duration_microseconds{handler="drop_series",quantile="0.5"} NaN
http_request_duration_microseconds{handler="drop_series",quantile="0.9"} NaN
http_request_duration_microseconds{handler="drop_series",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="drop_series"} 0
http_request_duration_microseconds_count{handler="drop_series"} 0
http_request_duration_microseconds{handler="federate",quantile="0.5"} NaN
http_request_duration_microseconds{handler="federate",quantile="0.9"} NaN
http_request_duration_microseconds{handler="federate",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="federate"} 0
http_request_duration_microseconds_count{handler="federate"} 0
http_request_duration_microseconds{handler="flags",quantile="0.5"} NaN
http_request_duration_microseconds{handler="flags",quantile="0.9"} NaN
http_request_duration_microseconds{handler="flags",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="flags"} 0
http_request_duration_microseconds_count{handler="flags"} 0
http_request_duration_microseconds{handler="graph",quantile="0.5"} NaN
http_request_duration_microseconds{handler="graph",quantile="0.9"} NaN
http_request_duration_microseconds{handler="graph",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="graph"} 17706.056999999997
http_request_duration_microseconds_count{handler="graph"} 18
http_request_duration_microseconds{handler="heap",quantile="0.5"} NaN
http_request_duration_microseconds{handler="heap",quantile="0.9"} NaN
http_request_duration_microseconds{handler="heap",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="heap"} 0
http_request_duration_microseconds_count{handler="heap"} 0
http_request_duration_microseconds{handler="label_values",quantile="0.5"} NaN
http_request_duration_microseconds{handler="label_values",quantile="0.9"} NaN
http_request_duration_microseconds{handler="label_values",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="label_values"} 120727.354
http_request_duration_microseconds_count{handler="label_values"} 128
http_request_duration_microseconds{handler="options",quantile="0.5"} NaN
http_request_duration_microseconds{handler="options",quantile="0.9"} NaN
http_request_duration_microseconds{handler="options",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="options"} 0
http_request_duration_microseconds_count{handler="options"} 0
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 6528.486
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 7499.182
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 7499.182
http_request_duration_microseconds_sum{handler="prometheus"} 23683.728
http_request_duration_microseconds_count{handler="prometheus"} 3
http_request_duration_microseconds{handler="query",quantile="0.5"} NaN
http_request_duration_microseconds{handler="query",quantile="0.9"} NaN
http_request_duration_microseconds{handler="query",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="query"} 298252.2080000001
http_request_duration_microseconds_count{handler="query"} 81
http_request_duration_microseconds{handler="query_range",quantile="0.5"} NaN
http_request_duration_microseconds{handler="query_range",quantile="0.9"} NaN
http_request_duration_microseconds{handler="query_range",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="query_range"} 1.442855006654999e+09
http_request_duration_microseconds_count{handler="query_range"} 19654
http_request_duration_microseconds{handler="rules",quantile="0.5"} NaN
http_request_duration_microseconds{handler="rules",quantile="0.9"} NaN
http_request_duration_microseconds{handler="rules",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="rules"} 1833.6779999999999
http_request_duration_microseconds_count{handler="rules"} 2
http_request_duration_microseconds{handler="series",quantile="0.5"} NaN
http_request_duration_microseconds{handler="series",quantile="0.9"} NaN
http_request_duration_microseconds{handler="series",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="series"} 0
http_request_duration_microseconds_count{handler="series"} 0
http_request_duration_microseconds{handler="static",quantile="0.5"} NaN
http_request_duration_microseconds{handler="static",quantile="0.9"} NaN
http_request_duration_microseconds{handler="static",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="static"} 185875.94199999995
http_request_duration_microseconds_count{handler="static"} 189
http_request_duration_microseconds{handler="status",quantile="0.5"} NaN
http_request_duration_microseconds{handler="status",quantile="0.9"} NaN
http_request_duration_microseconds{handler="status",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="status"} 851.926
http_request_duration_microseconds_count{handler="status"} 1
http_request_duration_microseconds{handler="targets",quantile="0.5"} NaN
http_request_duration_microseconds{handler="targets",quantile="0.9"} NaN
http_request_duration_microseconds{handler="targets",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="targets"} 168750.79700000002
http_request_duration_microseconds_count{handler="targets"} 31
http_request_duration_microseconds{handler="version",quantile="0.5"} NaN
http_request_duration_microseconds{handler="version",quantile="0.9"} NaN
http_request_duration_microseconds{handler="version",quantile="0.99"} NaN
http_request_duration_microseconds_sum{handler="version"} 0
http_request_duration_microseconds_count{handler="version"} 0
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="alertmanagers",quantile="0.5"} NaN
http_request_size_bytes{handler="alertmanagers",quantile="0.9"} NaN
http_request_size_bytes{handler="alertmanagers",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="alertmanagers"} 0
http_request_size_bytes_count{handler="alertmanagers"} 0
http_request_size_bytes{handler="alerts",quantile="0.5"} NaN
http_request_size_bytes{handler="alerts",quantile="0.9"} NaN
http_request_size_bytes{handler="alerts",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="alerts"} 934
http_request_size_bytes_count{handler="alerts"} 1
http_request_size_bytes{handler="config",quantile="0.5"} NaN
http_request_size_bytes{handler="config",quantile="0.9"} NaN
http_request_size_bytes{handler="config",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="config"} 0
http_request_size_bytes_count{handler="config"} 0
http_request_size_bytes{handler="consoles",quantile="0.5"} NaN
http_request_size_bytes{handler="consoles",quantile="0.9"} NaN
http_request_size_bytes{handler="consoles",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="consoles"} 0
http_request_size_bytes_count{handler="consoles"} 0
http_request_size_bytes{handler="drop_series",quantile="0.5"} NaN
http_request_size_bytes{handler="drop_series",quantile="0.9"} NaN
http_request_size_bytes{handler="drop_series",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="drop_series"} 0
http_request_size_bytes_count{handler="drop_series"} 0
http_request_size_bytes{handler="federate",quantile="0.5"} NaN
http_request_size_bytes{handler="federate",quantile="0.9"} NaN
http_request_size_bytes{handler="federate",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="federate"} 0
http_request_size_bytes_count{handler="federate"} 0
http_request_size_bytes{handler="flags",quantile="0.5"} NaN
http_request_size_bytes{handler="flags",quantile="0.9"} NaN
http_request_size_bytes{handler="flags",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="flags"} 0
http_request_size_bytes_count{handler="flags"} 0
http_request_size_bytes{handler="graph",quantile="0.5"} NaN
http_request_size_bytes{handler="graph",quantile="0.9"} NaN
http_request_size_bytes{handler="graph",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="graph"} 23820
http_request_size_bytes_count{handler="graph"} 18
http_request_size_bytes{handler="heap",quantile="0.5"} NaN
http_request_size_bytes{handler="heap",quantile="0.9"} NaN
http_request_size_bytes{handler="heap",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="heap"} 0
http_request_size_bytes_count{handler="heap"} 0
http_request_size_bytes{handler="label_values",quantile="0.5"} NaN
http_request_size_bytes{handler="label_values",quantile="0.9"} NaN
http_request_size_bytes{handler="label_values",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="label_values"} 86780
http_request_size_bytes_count{handler="label_values"} 128
http_request_size_bytes{handler="options",quantile="0.5"} NaN
http_request_size_bytes{handler="options",quantile="0.9"} NaN
http_request_size_bytes{handler="options",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="options"} 0
http_request_size_bytes_count{handler="options"} 0
http_request_size_bytes{handler="prometheus",quantile="0.5"} 63
http_request_size_bytes{handler="prometheus",quantile="0.9"} 63
http_request_size_bytes{handler="prometheus",quantile="0.99"} 63
http_request_size_bytes_sum{handler="prometheus"} 189
http_request_size_bytes_count{handler="prometheus"} 3
http_request_size_bytes{handler="query",quantile="0.5"} NaN
http_request_size_bytes{handler="query",quantile="0.9"} NaN
http_request_size_bytes{handler="query",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="query"} 130966
http_request_size_bytes_count{handler="query"} 81
http_request_size_bytes{handler="query_range",quantile="0.5"} NaN
http_request_size_bytes{handler="query_range",quantile="0.9"} NaN
http_request_size_bytes{handler="query_range",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="query_range"} 1.2188481e+07
http_request_size_bytes_count{handler="query_range"} 19654
http_request_size_bytes{handler="rules",quantile="0.5"} NaN
http_request_size_bytes{handler="rules",quantile="0.9"} NaN
http_request_size_bytes{handler="rules",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="rules"} 2014
http_request_size_bytes_count{handler="rules"} 2
http_request_size_bytes{handler="series",quantile="0.5"} NaN
http_request_size_bytes{handler="series",quantile="0.9"} NaN
http_request_size_bytes{handler="series",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="series"} 0
http_request_size_bytes_count{handler="series"} 0
http_request_size_bytes{handler="static",quantile="0.5"} NaN
http_request_size_bytes{handler="static",quantile="0.9"} NaN
http_request_size_bytes{handler="static",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="static"} 269501
http_request_size_bytes_count{handler="static"} 189
http_request_size_bytes{handler="status",quantile="0.5"} NaN
http_request_size_bytes{handler="status",quantile="0.9"} NaN
http_request_size_bytes{handler="status",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="status"} 1544
http_request_size_bytes_count{handler="status"} 1
http_request_size_bytes{handler="targets",quantile="0.5"} NaN
http_request_size_bytes{handler="targets",quantile="0.9"} NaN
http_request_size_bytes{handler="targets",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="targets"} 47961
http_request_size_bytes_count{handler="targets"} 31
http_request_size_bytes{handler="version",quantile="0.5"} NaN
http_request_size_bytes{handler="version",quantile="0.9"} NaN
http_request_size_bytes{handler="version",quantile="0.99"} NaN
http_request_size_bytes_sum{handler="version"} 0
http_request_size_bytes_count{handler="version"} 0
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="alerts",method="get"} 1
http_requests_total{code="200",handler="graph",method="get"} 18
http_requests_total{code="200",handler="label_values",method="get"} 128
http_requests_total{code="200",handler="prometheus",method="get"} 3
http_requests_total{code="200",handler="query",method="get"} 77
http_requests_total{code="200",handler="query_range",method="get"} 19551
http_requests_total{code="200",handler="rules",method="get"} 2
http_requests_total{code="200",handler="static",method="get"} 188
http_requests_total{code="200",handler="status",method="get"} 1
http_requests_total{code="200",handler="targets",method="get"} 31
http_requests_total{code="304",handler="static",method="get"} 1
http_requests_total{code="400",handler="query",method="get"} 4
http_requests_total{code="400",handler="query_range",method="get"} 24
http_requests_total{code="503",handler="query_range",method="get"} 79
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="alertmanagers",quantile="0.5"} NaN
http_response_size_bytes{handler="alertmanagers",quantile="0.9"} NaN
http_response_size_bytes{handler="alertmanagers",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="alertmanagers"} 0
http_response_size_bytes_count{handler="alertmanagers"} 0
http_response_size_bytes{handler="alerts",quantile="0.5"} NaN
http_response_size_bytes{handler="alerts",quantile="0.9"} NaN
http_response_size_bytes{handler="alerts",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="alerts"} 7115
http_response_size_bytes_count{handler="alerts"} 1
http_response_size_bytes{handler="config",quantile="0.5"} NaN
http_response_size_bytes{handler="config",quantile="0.9"} NaN
http_response_size_bytes{handler="config",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="config"} 0
http_response_size_bytes_count{handler="config"} 0
http_response_size_bytes{handler="consoles",quantile="0.5"} NaN
http_response_size_bytes{handler="consoles",quantile="0.9"} NaN
http_response_size_bytes{handler="consoles",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="consoles"} 0
http_response_size_bytes_count{handler="consoles"} 0
http_response_size_bytes{handler="drop_series",quantile="0.5"} NaN
http_response_size_bytes{handler="drop_series",quantile="0.9"} NaN
http_response_size_bytes{handler="drop_series",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="drop_series"} 0
http_response_size_bytes_count{handler="drop_series"} 0
http_response_size_bytes{handler="federate",quantile="0.5"} NaN
http_response_size_bytes{handler="federate",quantile="0.9"} NaN
http_response_size_bytes{handler="federate",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="federate"} 0
http_response_size_bytes_count{handler="federate"} 0
http_response_size_bytes{handler="flags",quantile="0.5"} NaN
http_response_size_bytes{handler="flags",quantile="0.9"} NaN
http_response_size_bytes{handler="flags",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="flags"} 0
http_response_size_bytes_count{handler="flags"} 0
http_response_size_bytes{handler="graph",quantile="0.5"} NaN
http_response_size_bytes{handler="graph",quantile="0.9"} NaN
http_response_size_bytes{handler="graph",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="graph"} 82260
http_response_size_bytes_count{handler="graph"} 18
http_response_size_bytes{handler="heap",quantile="0.5"} NaN
http_response_size_bytes{handler="heap",quantile="0.9"} NaN
http_response_size_bytes{handler="heap",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="heap"} 0
http_response_size_bytes_count{handler="heap"} 0
http_response_size_bytes{handler="label_values",quantile="0.5"} NaN
http_response_size_bytes{handler="label_values",quantile="0.9"} NaN
http_response_size_bytes{handler="label_values",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="label_values"} 336265
http_response_size_bytes_count{handler="label_values"} 128
http_response_size_bytes{handler="options",quantile="0.5"} NaN
http_response_size_bytes{handler="options",quantile="0.9"} NaN
http_response_size_bytes{handler="options",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="options"} 0
http_response_size_bytes_count{handler="options"} 0
http_response_size_bytes{handler="prometheus",quantile="0.5"} 61882
http_response_size_bytes{handler="prometheus",quantile="0.9"} 61974
http_response_size_bytes{handler="prometheus",quantile="0.99"} 61974
http_response_size_bytes_sum{handler="prometheus"} 185840
http_response_size_bytes_count{handler="prometheus"} 3
http_response_size_bytes{handler="query",quantile="0.5"} NaN
http_response_size_bytes{handler="query",quantile="0.9"} NaN
http_response_size_bytes{handler="query",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="query"} 82447
http_response_size_bytes_count{handler="query"} 81
http_response_size_bytes{handler="query_range",quantile="0.5"} NaN
http_response_size_bytes{handler="query_range",quantile="0.9"} NaN
http_response_size_bytes{handler="query_range",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="query_range"} 1.52711848e+08
http_response_size_bytes_count{handler="query_range"} 19654
http_response_size_bytes{handler="rules",quantile="0.5"} NaN
http_response_size_bytes{handler="rules",quantile="0.9"} NaN
http_response_size_bytes{handler="rules",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="rules"} 8868
http_response_size_bytes_count{handler="rules"} 2
http_response_size_bytes{handler="series",quantile="0.5"} NaN
http_response_size_bytes{handler="series",quantile="0.9"} NaN
http_response_size_bytes{handler="series",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="series"} 0
http_response_size_bytes_count{handler="series"} 0
http_response_size_bytes{handler="static",quantile="0.5"} NaN
http_response_size_bytes{handler="static",quantile="0.9"} NaN
http_response_size_bytes{handler="static",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="static"} 8.041272e+06
http_response_size_bytes_count{handler="static"} 189
http_response_size_bytes{handler="status",quantile="0.5"} NaN
http_response_size_bytes{handler="status",quantile="0.9"} NaN
http_response_size_bytes{handler="status",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="status"} 4193
http_response_size_bytes_count{handler="status"} 1
http_response_size_bytes{handler="targets",quantile="0.5"} NaN
http_response_size_bytes{handler="targets",quantile="0.9"} NaN
http_response_size_bytes{handler="targets",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="targets"} 1.527706e+06
http_response_size_bytes_count{handler="targets"} 31
http_response_size_bytes{handler="version",quantile="0.5"} NaN
http_response_size_bytes{handler="version",quantile="0.9"} NaN
http_response_size_bytes{handler="version",quantile="0.99"} NaN
http_response_size_bytes_sum{handler="version"} 0
http_response_size_bytes_count{handler="version"} 0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 18965.12
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 35
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.016768e+09
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.50552077978e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 2.407985152e+09
# HELP prometheus_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which prometheus was built.
# TYPE prometheus_build_info gauge
prometheus_build_info{branch="master",goversion="go1.8.3",revision="3afb3fffa3a29c3de865e1172fb740442e9d0133",version="1.7.1"} 1
# HELP prometheus_config_last_reload_success_timestamp_seconds Timestamp of the last successful configuration reload.
# TYPE prometheus_config_last_reload_success_timestamp_seconds gauge
prometheus_config_last_reload_success_timestamp_seconds 1.50552078e+09
# HELP prometheus_config_last_reload_successful Whether the last configuration reload attempt was successful.
# TYPE prometheus_config_last_reload_successful gauge
prometheus_config_last_reload_successful 1
# HELP prometheus_engine_queries The current number of queries being executed or waiting.
# TYPE prometheus_engine_queries gauge
prometheus_engine_queries 0
# HELP prometheus_engine_queries_concurrent_max The max number of concurrent queries.
# TYPE prometheus_engine_queries_concurrent_max gauge
prometheus_engine_queries_concurrent_max 20
# HELP prometheus_engine_query_duration_seconds Query timings
# TYPE prometheus_engine_query_duration_seconds summary
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.5"} 2.0081e-05
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.9"} 9.2323e-05
prometheus_engine_query_duration_seconds{slice="inner_eval",quantile="0.99"} 9.9279e-05
prometheus_engine_query_duration_seconds_sum{slice="inner_eval"} 1053.892852791008
prometheus_engine_query_duration_seconds_count{slice="inner_eval"} 108821
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.5"} 0.000180028
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.9"} 0.000323202
prometheus_engine_query_duration_seconds{slice="prepare_time",quantile="0.99"} 0.000445855
prometheus_engine_query_duration_seconds_sum{slice="prepare_time"} 45.68969470000014
prometheus_engine_query_duration_seconds_count{slice="prepare_time"} 108899
prometheus_engine_query_duration_seconds{slice="result_append",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="result_append",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="result_append",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="result_append"} 0.031072485000000146
prometheus_engine_query_duration_seconds_count{slice="result_append"} 19551
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.5"} NaN
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.9"} NaN
prometheus_engine_query_duration_seconds{slice="result_sort",quantile="0.99"} NaN
prometheus_engine_query_duration_seconds_sum{slice="result_sort"} 0.511905827999998
prometheus_engine_query_duration_seconds_count{slice="result_sort"} 19551
# HELP prometheus_evaluator_duration_seconds The duration of rule group evaluations.
# TYPE prometheus_evaluator_duration_seconds summary
prometheus_evaluator_duration_seconds{quantile="0.01"} 0.000551276
prometheus_evaluator_duration_seconds{quantile="0.05"} 0.000551276
prometheus_evaluator_duration_seconds{quantile="0.5"} 0.000592729
prometheus_evaluator_duration_seconds{quantile="0.9"} 0.000964238
prometheus_evaluator_duration_seconds{quantile="0.99"} 0.001121588
prometheus_evaluator_duration_seconds_sum 25.757426926999937
prometheus_evaluator_duration_seconds_count 22298
# HELP prometheus_evaluator_iterations_missed_total The total number of rule group evaluations missed due to slow rule group evaluation.
# TYPE prometheus_evaluator_iterations_missed_total counter
prometheus_evaluator_iterations_missed_total 0
# HELP prometheus_evaluator_iterations_skipped_total The total number of rule group evaluations skipped due to throttled metric storage.
# TYPE prometheus_evaluator_iterations_skipped_total counter
prometheus_evaluator_iterations_skipped_total 0
# HELP prometheus_evaluator_iterations_total The total number of scheduled rule group evaluations, whether executed, missed or skipped.
# TYPE prometheus_evaluator_iterations_total counter
prometheus_evaluator_iterations_total 22298
# HELP prometheus_local_storage_checkpoint_duration_seconds The duration in seconds taken for checkpointing open chunks and chunks yet to be persisted
# TYPE prometheus_local_storage_checkpoint_duration_seconds summary
prometheus_local_storage_checkpoint_duration_seconds_sum 787.3781688399993
prometheus_local_storage_checkpoint_duration_seconds_count 1193
# HELP prometheus_local_storage_checkpoint_last_duration_seconds The duration in seconds it took to last checkpoint open chunks and chunks yet to be persisted.
# TYPE prometheus_local_storage_checkpoint_last_duration_seconds gauge
prometheus_local_storage_checkpoint_last_duration_seconds 0.653727443
# HELP prometheus_local_storage_checkpoint_last_size_bytes The size of the last checkpoint of open chunks and chunks yet to be persisted
# TYPE prometheus_local_storage_checkpoint_last_size_bytes gauge
prometheus_local_storage_checkpoint_last_size_bytes 4.9682061e+07
# HELP prometheus_local_storage_checkpoint_series_chunks_written The number of chunk written per series while checkpointing open chunks and chunks yet to be persisted.
# TYPE prometheus_local_storage_checkpoint_series_chunks_written summary
prometheus_local_storage_checkpoint_series_chunks_written_sum 1.63289529e+08
prometheus_local_storage_checkpoint_series_chunks_written_count 6.4152119e+07
# HELP prometheus_local_storage_checkpointing 1 if the storage is checkpointing, 0 otherwise.
# TYPE prometheus_local_storage_checkpointing gauge
prometheus_local_storage_checkpointing 0
# HELP prometheus_local_storage_chunk_ops_total The total number of chunk operations by their type.
# TYPE prometheus_local_storage_chunk_ops_total counter
prometheus_local_storage_chunk_ops_total{type="clone"} 17016
prometheus_local_storage_chunk_ops_total{type="create"} 1.693707e+06
prometheus_local_storage_chunk_ops_total{type="evict"} 582177
prometheus_local_storage_chunk_ops_total{type="load"} 58393
prometheus_local_storage_chunk_ops_total{type="persist"} 815617
prometheus_local_storage_chunk_ops_total{type="pin"} 6.77553e+06
prometheus_local_storage_chunk_ops_total{type="transcode"} 1.203316e+06
prometheus_local_storage_chunk_ops_total{type="unpin"} 6.77553e+06
# HELP prometheus_local_storage_chunkdesc_ops_total The total number of chunk descriptor operations by their type.
# TYPE prometheus_local_storage_chunkdesc_ops_total counter
prometheus_local_storage_chunkdesc_ops_total{type="evict"} 106478
prometheus_local_storage_chunkdesc_ops_total{type="load"} 224253
# HELP prometheus_local_storage_chunks_to_persist The current number of chunks waiting for persistence.
# TYPE prometheus_local_storage_chunks_to_persist gauge
prometheus_local_storage_chunks_to_persist 829899
# HELP prometheus_local_storage_fingerprint_mappings_total The total number of fingerprints being mapped to avoid collisions.
# TYPE prometheus_local_storage_fingerprint_mappings_total counter
prometheus_local_storage_fingerprint_mappings_total 0
# HELP prometheus_local_storage_inconsistencies_total A counter incremented each time an inconsistency in the local storage is detected. If this is greater zero, restart the server as soon as possible.
# TYPE prometheus_local_storage_inconsistencies_total counter
prometheus_local_storage_inconsistencies_total 0
# HELP prometheus_local_storage_indexing_batch_duration_seconds Quantiles for batch indexing duration in seconds.
# TYPE prometheus_local_storage_indexing_batch_duration_seconds summary
prometheus_local_storage_indexing_batch_duration_seconds{quantile="0.5"} NaN
prometheus_local_storage_indexing_batch_duration_seconds{quantile="0.9"} NaN
prometheus_local_storage_indexing_batch_duration_seconds{quantile="0.99"} NaN
prometheus_local_storage_indexing_batch_duration_seconds_sum 3.6118764529999967
prometheus_local_storage_indexing_batch_duration_seconds_count 879
# HELP prometheus_local_storage_indexing_batch_sizes Quantiles for indexing batch sizes (number of metrics per batch).
# TYPE prometheus_local_storage_indexing_batch_sizes summary
prometheus_local_storage_indexing_batch_sizes{quantile="0.5"} NaN
prometheus_local_storage_indexing_batch_sizes{quantile="0.9"} NaN
prometheus_local_storage_indexing_batch_sizes{quantile="0.99"} NaN
prometheus_local_storage_indexing_batch_sizes_sum 63104
prometheus_local_storage_indexing_batch_sizes_count 879
# HELP prometheus_local_storage_indexing_queue_capacity The capacity of the indexing queue.
# TYPE prometheus_local_storage_indexing_queue_capacity gauge
prometheus_local_storage_indexing_queue_capacity 262144
# HELP prometheus_local_storage_indexing_queue_length The number of metrics waiting to be indexed.
# TYPE prometheus_local_storage_indexing_queue_length gauge
prometheus_local_storage_indexing_queue_length 0
# HELP prometheus_local_storage_ingested_samples_total The total number of samples ingested.
# TYPE prometheus_local_storage_ingested_samples_total counter
prometheus_local_storage_ingested_samples_total 6.967072e+08
# HELP prometheus_local_storage_maintain_series_duration_seconds The duration in seconds it took to perform maintenance on a series.
# TYPE prometheus_local_storage_maintain_series_duration_seconds summary
prometheus_local_storage_maintain_series_duration_seconds{location="archived",quantile="0.5"} NaN
prometheus_local_storage_maintain_series_duration_seconds{location="archived",quantile="0.9"} NaN
prometheus_local_storage_maintain_series_duration_seconds{location="archived",quantile="0.99"} NaN
prometheus_local_storage_maintain_series_duration_seconds_sum{location="archived"} 0
prometheus_local_storage_maintain_series_duration_seconds_count{location="archived"} 0
prometheus_local_storage_maintain_series_duration_seconds{location="memory",quantile="0.5"} NaN
prometheus_local_storage_maintain_series_duration_seconds{location="memory",quantile="0.9"} NaN
prometheus_local_storage_maintain_series_duration_seconds{location="memory",quantile="0.99"} NaN
prometheus_local_storage_maintain_series_duration_seconds_sum{location="memory"} 4937.689572514285
prometheus_local_storage_maintain_series_duration_seconds_count{location="memory"} 591241
# HELP prometheus_local_storage_memory_chunkdescs The current number of chunk descriptors in memory.
# TYPE prometheus_local_storage_memory_chunkdescs gauge
prometheus_local_storage_memory_chunkdescs 1.819389e+06
# HELP prometheus_local_storage_memory_chunks The current number of chunks in memory. The number does not include cloned chunks (i.e. chunks without a descriptor).
# TYPE prometheus_local_storage_memory_chunks gauge
prometheus_local_storage_memory_chunks 1.169923e+06
# HELP prometheus_local_storage_memory_dirty_series The current number of series that would require a disk seek during crash recovery.
# TYPE prometheus_local_storage_memory_dirty_series gauge
prometheus_local_storage_memory_dirty_series 15708
# HELP prometheus_local_storage_memory_series The current number of series in memory.
# TYPE prometheus_local_storage_memory_series gauge
prometheus_local_storage_memory_series 55498
# HELP prometheus_local_storage_non_existent_series_matches_total How often a non-existent series was referred to during label matching or chunk preloading. This is an indication of outdated label indexes.
# TYPE prometheus_local_storage_non_existent_series_matches_total counter
prometheus_local_storage_non_existent_series_matches_total 0
# HELP prometheus_local_storage_open_head_chunks The current number of open head chunks.
# TYPE prometheus_local_storage_open_head_chunks gauge
prometheus_local_storage_open_head_chunks 48191
# HELP prometheus_local_storage_out_of_order_samples_total The total number of samples that were discarded because their timestamps were at or before the last received sample for a series.
# TYPE prometheus_local_storage_out_of_order_samples_total counter
prometheus_local_storage_out_of_order_samples_total{reason="multiple_values_for_timestamp"} 0
prometheus_local_storage_out_of_order_samples_total{reason="timestamp_out_of_order"} 0
# HELP prometheus_local_storage_persist_errors_total The total number of errors while writing to the persistence layer.
# TYPE prometheus_local_storage_persist_errors_total counter
prometheus_local_storage_persist_errors_total 0
# HELP prometheus_local_storage_persistence_urgency_score A score of urgency to persist chunks, 0 is least urgent, 1 most.
# TYPE prometheus_local_storage_persistence_urgency_score gauge
prometheus_local_storage_persistence_urgency_score 0.711
# HELP prometheus_local_storage_queued_chunks_to_persist_total The total number of chunks queued for persistence.
# TYPE prometheus_local_storage_queued_chunks_to_persist_total counter
prometheus_local_storage_queued_chunks_to_persist_total 1.612698e+06
# HELP prometheus_local_storage_rushed_mode 1 if the storage is in rushed mode, 0 otherwise.
# TYPE prometheus_local_storage_rushed_mode gauge
prometheus_local_storage_rushed_mode 0
# HELP prometheus_local_storage_series_chunks_persisted The number of chunks persisted per series.
# TYPE prometheus_local_storage_series_chunks_persisted histogram
prometheus_local_storage_series_chunks_persisted_bucket{le="1"} 306134
prometheus_local_storage_series_chunks_persisted_bucket{le="2"} 343845
prometheus_local_storage_series_chunks_persisted_bucket{le="4"} 377010
prometheus_local_storage_series_chunks_persisted_bucket{le="8"} 427560
prometheus_local_storage_series_chunks_persisted_bucket{le="16"} 428248
prometheus_local_storage_series_chunks_persisted_bucket{le="32"} 428248
prometheus_local_storage_series_chunks_persisted_bucket{le="64"} 428248
prometheus_local_storage_series_chunks_persisted_bucket{le="128"} 428248
prometheus_local_storage_series_chunks_persisted_bucket{le="+Inf"} 428248
prometheus_local_storage_series_chunks_persisted_sum 815617
prometheus_local_storage_series_chunks_persisted_count 428248
# HELP prometheus_local_storage_series_ops_total The total number of series operations by their type.
# TYPE prometheus_local_storage_series_ops_total counter
prometheus_local_storage_series_ops_total{type="archive"} 9535
prometheus_local_storage_series_ops_total{type="create"} 23946
prometheus_local_storage_series_ops_total{type="maintenance_in_archive"} 0
prometheus_local_storage_series_ops_total{type="maintenance_in_memory"} 591241
prometheus_local_storage_series_ops_total{type="purge_from_archive"} 0
prometheus_local_storage_series_ops_total{type="purge_from_memory"} 0
prometheus_local_storage_series_ops_total{type="purge_on_request"} 0
prometheus_local_storage_series_ops_total{type="quarantine_completed"} 0
prometheus_local_storage_series_ops_total{type="quarantine_dropped"} 0
prometheus_local_storage_series_ops_total{type="quarantine_failed"} 0
prometheus_local_storage_series_ops_total{type="unarchive"} 3413
# HELP prometheus_local_storage_started_dirty Whether the local storage was found to be dirty (and crash recovery occurred) during Prometheus startup.
# TYPE prometheus_local_storage_started_dirty gauge
prometheus_local_storage_started_dirty 1
# HELP prometheus_local_storage_target_heap_size_bytes The configured target heap size in bytes.
# TYPE prometheus_local_storage_target_heap_size_bytes gauge
prometheus_local_storage_target_heap_size_bytes 2.147483648e+09
# HELP prometheus_notifications_alertmanagers_discovered The number of alertmanagers discovered and active.
# TYPE prometheus_notifications_alertmanagers_discovered gauge
prometheus_notifications_alertmanagers_discovered 1
# HELP prometheus_notifications_dropped_total Total number of alerts dropped due to errors when sending to Alertmanager.
# TYPE prometheus_notifications_dropped_total counter
prometheus_notifications_dropped_total 0
# HELP prometheus_notifications_errors_total Total number of errors sending alert notifications.
# TYPE prometheus_notifications_errors_total counter
prometheus_notifications_errors_total{alertmanager="http://localhost:9093/api/v1/alerts"} 0
# HELP prometheus_notifications_latency_seconds Latency quantiles for sending alert notifications (not including dropped notifications).
# TYPE prometheus_notifications_latency_seconds summary
prometheus_notifications_latency_seconds{alertmanager="http://localhost:9093/api/v1/alerts",quantile="0.5"} NaN
prometheus_notifications_latency_seconds{alertmanager="http://localhost:9093/api/v1/alerts",quantile="0.9"} NaN
prometheus_notifications_latency_seconds{alertmanager="http://localhost:9093/api/v1/alerts",quantile="0.99"} NaN
prometheus_notifications_latency_seconds_sum{alertmanager="http://localhost:9093/api/v1/alerts"} 4.048206670000008
prometheus_notifications_latency_seconds_count{alertmanager="http://localhost:9093/api/v1/alerts"} 5792
# HELP prometheus_notifications_queue_capacity The capacity of the alert notifications queue.
# TYPE prometheus_notifications_queue_capacity gauge
prometheus_notifications_queue_capacity 10000
# HELP prometheus_notifications_queue_length The number of alert notifications in the queue.
# TYPE prometheus_notifications_queue_length gauge
prometheus_notifications_queue_length 0
# HELP prometheus_notifications_sent_total Total number of alerts successfully sent.
# TYPE prometheus_notifications_sent_total counter
prometheus_notifications_sent_total{alertmanager="http://localhost:9093/api/v1/alerts"} 12899
# HELP prometheus_rule_evaluation_duration_seconds The duration for a rule to execute.
# TYPE prometheus_rule_evaluation_duration_seconds summary
prometheus_rule_evaluation_duration_seconds{rule_type="alerting",quantile="0.5"} 0.000338486
prometheus_rule_evaluation_duration_seconds{rule_type="alerting",quantile="0.9"} 0.000587035
prometheus_rule_evaluation_duration_seconds{rule_type="alerting",quantile="0.99"} 0.000940654
prometheus_rule_evaluation_duration_seconds_sum{rule_type="alerting"} 44.07109911500028
prometheus_rule_evaluation_duration_seconds_count{rule_type="alerting"} 89192
# HELP prometheus_rule_evaluation_failures_total The total number of rule evaluation failures.
# TYPE prometheus_rule_evaluation_failures_total counter
prometheus_rule_evaluation_failures_total{rule_type="alerting"} 0
prometheus_rule_evaluation_failures_total{rule_type="recording"} 0
# HELP prometheus_sd_azure_refresh_duration_seconds The duration of a Azure-SD refresh in seconds.
# TYPE prometheus_sd_azure_refresh_duration_seconds summary
prometheus_sd_azure_refresh_duration_seconds{quantile="0.5"} NaN
prometheus_sd_azure_refresh_duration_seconds{quantile="0.9"} NaN
prometheus_sd_azure_refresh_duration_seconds{quantile="0.99"} NaN
prometheus_sd_azure_refresh_duration_seconds_sum 0
prometheus_sd_azure_refresh_duration_seconds_count 0
# HELP prometheus_sd_azure_refresh_failures_total Number of Azure-SD refresh failures.
# TYPE prometheus_sd_azure_refresh_failures_total counter
prometheus_sd_azure_refresh_failures_total 0
# HELP prometheus_sd_consul_rpc_duration_seconds The duration of a Consul RPC call in seconds.
# TYPE prometheus_sd_consul_rpc_duration_seconds summary
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.5"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.9"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="service",endpoint="catalog",quantile="0.99"} NaN
prometheus_sd_consul_rpc_duration_seconds_sum{call="service",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds_count{call="service",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.5"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.9"} NaN
prometheus_sd_consul_rpc_duration_seconds{call="services",endpoint="catalog",quantile="0.99"} NaN
prometheus_sd_consul_rpc_duration_seconds_sum{call="services",endpoint="catalog"} 0
prometheus_sd_consul_rpc_duration_seconds_count{call="services",endpoint="catalog"} 0
# HELP prometheus_sd_consul_rpc_failures_total The number of Consul RPC call failures.
# TYPE prometheus_sd_consul_rpc_failures_total counter
prometheus_sd_consul_rpc_failures_total 0
# HELP prometheus_sd_dns_lookup_failures_total The number of DNS-SD lookup failures.
# TYPE prometheus_sd_dns_lookup_failures_total counter
prometheus_sd_dns_lookup_failures_total 0
# HELP prometheus_sd_dns_lookups_total The number of DNS-SD lookups.
# TYPE prometheus_sd_dns_lookups_total counter
prometheus_sd_dns_lookups_total 0
# HELP prometheus_sd_ec2_refresh_duration_seconds The duration of a EC2-SD refresh in seconds.
# TYPE prometheus_sd_ec2_refresh_duration_seconds summary
prometheus_sd_ec2_refresh_duration_seconds{quantile="0.5"} 0.429769719
prometheus_sd_ec2_refresh_duration_seconds{quantile="0.9"} 0.453263567
prometheus_sd_ec2_refresh_duration_seconds{quantile="0.99"} 0.469643112
prometheus_sd_ec2_refresh_duration_seconds_sum 57807.78184339297
prometheus_sd_ec2_refresh_duration_seconds_count 122639
# HELP prometheus_sd_ec2_refresh_failures_total The number of EC2-SD scrape failures.
# TYPE prometheus_sd_ec2_refresh_failures_total counter
prometheus_sd_ec2_refresh_failures_total 3
# HELP prometheus_sd_file_read_errors_total The number of File-SD read errors.
# TYPE prometheus_sd_file_read_errors_total counter
prometheus_sd_file_read_errors_total 0
# HELP prometheus_sd_file_scan_duration_seconds The duration of the File-SD scan in seconds.
# TYPE prometheus_sd_file_scan_duration_seconds summary
prometheus_sd_file_scan_duration_seconds{quantile="0.5"} NaN
prometheus_sd_file_scan_duration_seconds{quantile="0.9"} NaN
prometheus_sd_file_scan_duration_seconds{quantile="0.99"} NaN
prometheus_sd_file_scan_duration_seconds_sum 0
prometheus_sd_file_scan_duration_seconds_count 0
# HELP prometheus_sd_gce_refresh_duration The duration of a GCE-SD refresh in seconds.
# TYPE prometheus_sd_gce_refresh_duration summary
prometheus_sd_gce_refresh_duration{quantile="0.5"} NaN
prometheus_sd_gce_refresh_duration{quantile="0.9"} NaN
prometheus_sd_gce_refresh_duration{quantile="0.99"} NaN
prometheus_sd_gce_refresh_duration_sum 0
prometheus_sd_gce_refresh_duration_count 0
# HELP prometheus_sd_gce_refresh_failures_total The number of GCE-SD refresh failures.
# TYPE prometheus_sd_gce_refresh_failures_total counter
prometheus_sd_gce_refresh_failures_total 0
# HELP prometheus_sd_kubernetes_events_total The number of Kubernetes events handled.
# TYPE prometheus_sd_kubernetes_events_total counter
prometheus_sd_kubernetes_events_total{event="add",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="add",role="node"} 0
prometheus_sd_kubernetes_events_total{event="add",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="add",role="service"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="node"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="delete",role="service"} 0
prometheus_sd_kubernetes_events_total{event="update",role="endpoints"} 0
prometheus_sd_kubernetes_events_total{event="update",role="node"} 0
prometheus_sd_kubernetes_events_total{event="update",role="pod"} 0
prometheus_sd_kubernetes_events_total{event="update",role="service"} 0
# HELP prometheus_sd_marathon_refresh_duration_seconds The duration of a Marathon-SD refresh in seconds.
# TYPE prometheus_sd_marathon_refresh_duration_seconds summary
prometheus_sd_marathon_refresh_duration_seconds{quantile="0.5"} NaN
prometheus_sd_marathon_refresh_duration_seconds{quantile="0.9"} NaN
prometheus_sd_marathon_refresh_duration_seconds{quantile="0.99"} NaN
prometheus_sd_marathon_refresh_duration_seconds_sum 0
prometheus_sd_marathon_refresh_duration_seconds_count 0
# HELP prometheus_sd_marathon_refresh_failures_total The number of Marathon-SD refresh failures.
# TYPE prometheus_sd_marathon_refresh_failures_total counter
prometheus_sd_marathon_refresh_failures_total 0
# HELP prometheus_sd_openstack_refresh_duration_seconds The duration of an OpenStack-SD refresh in seconds.
# TYPE prometheus_sd_openstack_refresh_duration_seconds summary
prometheus_sd_openstack_refresh_duration_seconds{quantile="0.5"} NaN
prometheus_sd_openstack_refresh_duration_seconds{quantile="0.9"} NaN
prometheus_sd_openstack_refresh_duration_seconds{quantile="0.99"} NaN
prometheus_sd_openstack_refresh_duration_seconds_sum 0
prometheus_sd_openstack_refresh_duration_seconds_count 0
# HELP prometheus_sd_openstack_refresh_failures_total The number of OpenStack-SD scrape failures.
# TYPE prometheus_sd_openstack_refresh_failures_total counter
prometheus_sd_openstack_refresh_failures_total 0
# HELP prometheus_sd_triton_refresh_duration_seconds The duration of a Triton-SD refresh in seconds.
# TYPE prometheus_sd_triton_refresh_duration_seconds summary
prometheus_sd_triton_refresh_duration_seconds{quantile="0.5"} NaN
prometheus_sd_triton_refresh_duration_seconds{quantile="0.9"} NaN
prometheus_sd_triton_refresh_duration_seconds{quantile="0.99"} NaN
prometheus_sd_triton_refresh_duration_seconds_sum 0
prometheus_sd_triton_refresh_duration_seconds_count 0
# HELP prometheus_sd_triton_refresh_failures_total The number of Triton-SD scrape failures.
# TYPE prometheus_sd_triton_refresh_failures_total counter
prometheus_sd_triton_refresh_failures_total 0
# HELP prometheus_target_interval_length_seconds Actual intervals between scrapes.
# TYPE prometheus_target_interval_length_seconds summary
prometheus_target_interval_length_seconds{interval="30s",quantile="0.01"} 29.995216834
prometheus_target_interval_length_seconds{interval="30s",quantile="0.05"} 29.999920687
prometheus_target_interval_length_seconds{interval="30s",quantile="0.5"} 30.00001506
prometheus_target_interval_length_seconds{interval="30s",quantile="0.9"} 30.000078212
prometheus_target_interval_length_seconds{interval="30s",quantile="0.99"} 30.004683549
prometheus_target_interval_length_seconds_sum{interval="30s"} 1.8182410244003933e+07
prometheus_target_interval_length_seconds_count{interval="30s"} 606080
# HELP prometheus_target_scrape_pool_sync_total Total number of syncs that were executed on a scrape pool.
# TYPE prometheus_target_scrape_pool_sync_total counter
prometheus_target_scrape_pool_sync_total{scrape_job="hermes_ingester_py"} 1
prometheus_target_scrape_pool_sync_total{scrape_job="tclips"} 11148
prometheus_target_scrape_pool_sync_total{scrape_job="tjenkins-exp"} 11149
prometheus_target_scrape_pool_sync_total{scrape_job="tlabels-exp"} 11149
prometheus_target_scrape_pool_sync_total{scrape_job="tlabels-web"} 11148
prometheus_target_scrape_pool_sync_total{scrape_job="ttel-exp"} 11149
prometheus_target_scrape_pool_sync_total{scrape_job="ttel-gw"} 1
prometheus_target_scrape_pool_sync_total{scrape_job="ttel-proc"} 11148
prometheus_target_scrape_pool_sync_total{scrape_job="ttel-proc-images"} 11149
prometheus_target_scrape_pool_sync_total{scrape_job="ttel-proc-snap"} 11149
prometheus_target_scrape_pool_sync_total{scrape_job="ttel-tail"} 11149
prometheus_target_scrape_pool_sync_total{scrape_job="ttel-tail-images"} 11149
prometheus_target_scrape_pool_sync_total{scrape_job="ttel-tfiles"} 11149
# HELP prometheus_target_scrapes_exceeded_sample_limit_total Total number of scrapes that hit the sample limit and were rejected.
# TYPE prometheus_target_scrapes_exceeded_sample_limit_total counter
prometheus_target_scrapes_exceeded_sample_limit_total 0
# HELP prometheus_target_skipped_scrapes_total Total number of scrapes that were skipped because the metric storage was throttled.
# TYPE prometheus_target_skipped_scrapes_total counter
prometheus_target_skipped_scrapes_total 0
# HELP prometheus_target_sync_length_seconds Actual interval to sync the scrape pool.
# TYPE prometheus_target_sync_length_seconds summary
prometheus_target_sync_length_seconds{scrape_job="hermes_ingester_py",quantile="0.01"} NaN
prometheus_target_sync_length_seconds{scrape_job="hermes_ingester_py",quantile="0.05"} NaN
prometheus_target_sync_length_seconds{scrape_job="hermes_ingester_py",quantile="0.5"} NaN
prometheus_target_sync_length_seconds{scrape_job="hermes_ingester_py",quantile="0.9"} NaN
prometheus_target_sync_length_seconds{scrape_job="hermes_ingester_py",quantile="0.99"} NaN
prometheus_target_sync_length_seconds_sum{scrape_job="hermes_ingester_py"} 7.4019e-05
prometheus_target_sync_length_seconds_count{scrape_job="hermes_ingester_py"} 1
prometheus_target_sync_length_seconds{scrape_job="tclips",quantile="0.01"} 0.000338448
prometheus_target_sync_length_seconds{scrape_job="tclips",quantile="0.05"} 0.000338448
prometheus_target_sync_length_seconds{scrape_job="tclips",quantile="0.5"} 0.000352895
prometheus_target_sync_length_seconds{scrape_job="tclips",quantile="0.9"} 0.000371729
prometheus_target_sync_length_seconds{scrape_job="tclips",quantile="0.99"} 0.000371729
prometheus_target_sync_length_seconds_sum{scrape_job="tclips"} 8.621404031999974
prometheus_target_sync_length_seconds_count{scrape_job="tclips"} 11148
prometheus_target_sync_length_seconds{scrape_job="tjenkins-exp",quantile="0.01"} 0.00030578
prometheus_target_sync_length_seconds{scrape_job="tjenkins-exp",quantile="0.05"} 0.00030578
prometheus_target_sync_length_seconds{scrape_job="tjenkins-exp",quantile="0.5"} 0.000336116
prometheus_target_sync_length_seconds{scrape_job="tjenkins-exp",quantile="0.9"} 0.000356755
prometheus_target_sync_length_seconds{scrape_job="tjenkins-exp",quantile="0.99"} 0.000356755
prometheus_target_sync_length_seconds_sum{scrape_job="tjenkins-exp"} 8.034462578000008
prometheus_target_sync_length_seconds_count{scrape_job="tjenkins-exp"} 11149
prometheus_target_sync_length_seconds{scrape_job="tlabels-exp",quantile="0.01"} 0.000351241
prometheus_target_sync_length_seconds{scrape_job="tlabels-exp",quantile="0.05"} 0.000351241
prometheus_target_sync_length_seconds{scrape_job="tlabels-exp",quantile="0.5"} 0.000365992
prometheus_target_sync_length_seconds{scrape_job="tlabels-exp",quantile="0.9"} 0.000387015
prometheus_target_sync_length_seconds{scrape_job="tlabels-exp",quantile="0.99"} 0.000387015
prometheus_target_sync_length_seconds_sum{scrape_job="tlabels-exp"} 8.133431119999985
prometheus_target_sync_length_seconds_count{scrape_job="tlabels-exp"} 11149
prometheus_target_sync_length_seconds{scrape_job="tlabels-web",quantile="0.01"} 0.000343228
prometheus_target_sync_length_seconds{scrape_job="tlabels-web",quantile="0.05"} 0.000343228
prometheus_target_sync_length_seconds{scrape_job="tlabels-web",quantile="0.5"} 0.000355265
prometheus_target_sync_length_seconds{scrape_job="tlabels-web",quantile="0.9"} 0.000372067
prometheus_target_sync_length_seconds{scrape_job="tlabels-web",quantile="0.99"} 0.000372067
prometheus_target_sync_length_seconds_sum{scrape_job="tlabels-web"} 8.402049760999988
prometheus_target_sync_length_seconds_count{scrape_job="tlabels-web"} 11148
prometheus_target_sync_length_seconds{scrape_job="ttel-exp",quantile="0.01"} 0.000409275
prometheus_target_sync_length_seconds{scrape_job="ttel-exp",quantile="0.05"} 0.000409275
prometheus_target_sync_length_seconds{scrape_job="ttel-exp",quantile="0.5"} 0.000428158
prometheus_target_sync_length_seconds{scrape_job="ttel-exp",quantile="0.9"} 0.000450352
prometheus_target_sync_length_seconds{scrape_job="ttel-exp",quantile="0.99"} 0.000450352
prometheus_target_sync_length_seconds_sum{scrape_job="ttel-exp"} 9.627508978000007
prometheus_target_sync_length_seconds_count{scrape_job="ttel-exp"} 11149
prometheus_target_sync_length_seconds{scrape_job="ttel-gw",quantile="0.01"} NaN
prometheus_target_sync_length_seconds{scrape_job="ttel-gw",quantile="0.05"} NaN
prometheus_target_sync_length_seconds{scrape_job="ttel-gw",quantile="0.5"} NaN
prometheus_target_sync_length_seconds{scrape_job="ttel-gw",quantile="0.9"} NaN
prometheus_target_sync_length_seconds{scrape_job="ttel-gw",quantile="0.99"} NaN
prometheus_target_sync_length_seconds_sum{scrape_job="ttel-gw"} 3.5626e-05
prometheus_target_sync_length_seconds_count{scrape_job="ttel-gw"} 1
prometheus_target_sync_length_seconds{scrape_job="ttel-proc",quantile="0.01"} 0.000307688
prometheus_target_sync_length_seconds{scrape_job="ttel-proc",quantile="0.05"} 0.000307688
prometheus_target_sync_length_seconds{scrape_job="ttel-proc",quantile="0.5"} 0.000337855
prometheus_target_sync_length_seconds{scrape_job="ttel-proc",quantile="0.9"} 0.000359992
prometheus_target_sync_length_seconds{scrape_job="ttel-proc",quantile="0.99"} 0.000359992
prometheus_target_sync_length_seconds_sum{scrape_job="ttel-proc"} 8.491405452999976
prometheus_target_sync_length_seconds_count{scrape_job="ttel-proc"} 11148
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-images",quantile="0.01"} 0.000314498
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-images",quantile="0.05"} 0.000314498
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-images",quantile="0.5"} 0.000333424
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-images",quantile="0.9"} 0.000345796
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-images",quantile="0.99"} 0.000345796
prometheus_target_sync_length_seconds_sum{scrape_job="ttel-proc-images"} 8.022680775999996
prometheus_target_sync_length_seconds_count{scrape_job="ttel-proc-images"} 11149
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-snap",quantile="0.01"} 0.000324282
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-snap",quantile="0.05"} 0.000324282
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-snap",quantile="0.5"} 0.000340456
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-snap",quantile="0.9"} 0.000361183
prometheus_target_sync_length_seconds{scrape_job="ttel-proc-snap",quantile="0.99"} 0.000361183
prometheus_target_sync_length_seconds_sum{scrape_job="ttel-proc-snap"} 7.214494955999999
prometheus_target_sync_length_seconds_count{scrape_job="ttel-proc-snap"} 11149
prometheus_target_sync_length_seconds{scrape_job="ttel-tail",quantile="0.01"} 0.000317922
prometheus_target_sync_length_seconds{scrape_job="ttel-tail",quantile="0.05"} 0.000317922
prometheus_target_sync_length_seconds{scrape_job="ttel-tail",quantile="0.5"} 0.000335761
prometheus_target_sync_length_seconds{scrape_job="ttel-tail",quantile="0.9"} 0.000352961
prometheus_target_sync_length_seconds{scrape_job="ttel-tail",quantile="0.99"} 0.000352961
prometheus_target_sync_length_seconds_sum{scrape_job="ttel-tail"} 7.740362498000005
prometheus_target_sync_length_seconds_count{scrape_job="ttel-tail"} 11149
prometheus_target_sync_length_seconds{scrape_job="ttel-tail-images",quantile="0.01"} 0.000322534
prometheus_target_sync_length_seconds{scrape_job="ttel-tail-images",quantile="0.05"} 0.000322534
prometheus_target_sync_length_seconds{scrape_job="ttel-tail-images",quantile="0.5"} 0.000334813
prometheus_target_sync_length_seconds{scrape_job="ttel-tail-images",quantile="0.9"} 0.000338759
prometheus_target_sync_length_seconds{scrape_job="ttel-tail-images",quantile="0.99"} 0.000338759
prometheus_target_sync_length_seconds_sum{scrape_job="ttel-tail-images"} 7.664829292999984
prometheus_target_sync_length_seconds_count{scrape_job="ttel-tail-images"} 11149
prometheus_target_sync_length_seconds{scrape_job="ttel-tfiles",quantile="0.01"} 0.000343372
prometheus_target_sync_length_seconds{scrape_job="ttel-tfiles",quantile="0.05"} 0.000343372
prometheus_target_sync_length_seconds{scrape_job="ttel-tfiles",quantile="0.5"} 0.000356419
prometheus_target_sync_length_seconds{scrape_job="ttel-tfiles",quantile="0.9"} 0.000368897
prometheus_target_sync_length_seconds{scrape_job="ttel-tfiles",quantile="0.99"} 0.000368897
prometheus_target_sync_length_seconds_sum{scrape_job="ttel-tfiles"} 7.728078352000026
prometheus_target_sync_length_seconds_count{scrape_job="ttel-tfiles"} 11149
# HELP prometheus_treecache_watcher_goroutines The current number of watcher goroutines.
# TYPE prometheus_treecache_watcher_goroutines gauge
prometheus_treecache_watcher_goroutines 0
# HELP prometheus_treecache_zookeeper_failures_total The total number of ZooKeeper failures.
# TYPE prometheus_treecache_zookeeper_failures_total counter
prometheus_treecache_zookeeper_failures_total 0

[korovkin]

seems stuck like that for about 3 days (time="2017-09-19T01:13:22-07:00" ), even though it is fully functional (responds to all queries and requests)

[korovkin]

please help

[brian-brazil]

It makes more sense to ask questions like this on the prometheus-users mailing list rather than in a GitHub issue. On the mailing list, more people are available to potentially respond to your question, and the whole community can benefit from the answers provided.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.