Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fatal error: concurrent map read and map write (when using delete series) #4154

Closed
games130 opened this Issue May 11, 2018 · 10 comments

Comments

Assignees
No one assigned
Labels
component/local storage kind/bug
Projects
None yet
Milestone
No milestone
5 participants
@games130 @brian-brazil @gouthamve @codwu @krasi-georgiev
@games130
Copy link

games130 commented May 11, 2018

What did you do?
I run cronjob to delete metric data every 5min
curl -XPOST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]=heplify_caller&end=xxx'
xxx is current time minus 1 hour.

I am doing this because all my other data is stored for 6months but this particular metric, i only need latest 1 hour. So I will delete anything older than 1 hour.

What did you expect to see?
metrics older than 1 hour are deleted every 5min

What did you see instead? Under which circumstances?
The deletion of the metric works, but prometheus crash frequently. Attached below is the log
I tested by removing the cronjob and it runs without crashing.

Environment

  • System information:

Linux 3.10.0-693.11.1.el7.x86_64 x86_64
I am running docker for the prometheus

  • Prometheus version:
    version=2.2.1

  • Prometheus configuration file:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
  external_labels:
    monitor: docker-host-alpha
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
    scheme: http
    timeout: 10s
rule_files:
- /etc/prometheus/alert.rules
scrape_configs:
- job_name: nodeexporter
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:9100
- job_name: cadvisor
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:8080
- job_name: prometheus
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:9090
- job_name: heplify-server
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:9999
- job_name: horaclifix
  scrape_interval: 5s
  scrape_timeout: 5s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:8899
  • Logs:
level=info ts=2018-05-10T06:02:29.00073869Z caller=main.go:220 msg="Starting Prometheus" version="(version=2.2.1, branch=HEAD, revision=bc6058c81272a8d938c05e75607371284236aadc)"
level=info ts=2018-05-10T06:02:29.000861733Z caller=main.go:221 build_context="(go=go1.10, user=root@149e5b3f0829, date=20180314-14:15:45)"
level=info ts=2018-05-10T06:02:29.000908032Z caller=main.go:222 host_details="(Linux 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 localhost.localdomain (none))"
level=info ts=2018-05-10T06:02:29.000947309Z caller=main.go:223 fd_limits="(soft=65536, hard=65536)"
level=info ts=2018-05-10T06:02:29.004913825Z caller=web.go:382 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-05-10T06:02:29.004789674Z caller=main.go:504 msg="Starting TSDB ..."
level=info ts=2018-05-10T06:02:49.938233809Z caller=main.go:514 msg="TSDB started"
level=info ts=2018-05-10T06:02:49.938340714Z caller=main.go:588 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2018-05-10T06:02:49.941304795Z caller=main.go:491 msg="Server is ready to receive web requests."
level=info ts=2018-05-10T06:02:53.438975862Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1525910400000 maxt=1525917600000
level=info ts=2018-05-10T06:02:53.863568405Z caller=compact.go:393 component=tsdb msg="compact blocks" count=1 mint=1525917600000 maxt=1525924800000
fatal error: concurrent map read and map write

goroutine 42288 [running]:
runtime.throw(0x1b03839, 0x21)
        /usr/local/go/src/runtime/panic.go:619 +0x81 fp=0xc4bdece6c8 sp=0xc4bdece6a8 pc=0x42b451
runtime.mapaccess1_fast64(0x1916ca0, 0xc420bedb60, 0x1ca05, 0x16348c71c2b)
        /usr/local/go/src/runtime/hashmap_fast.go:101 +0x197 fp=0xc4bdece6f0 sp=0xc4bdece6c8 pc=0x40a7c7
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.memTombstones.Get(0xc420bedb60, 0x1ca05, 0xc4314df600, 0xc4314df620, 0x0, 0x0, 0xc4314df5e0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/tombstones.go:169 +0x41 fp=0xc4bdece720 sp=0xc4bdece6f0 pc=0x14b0a71
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*baseChunkSeries).Next(0xc43a114fc0, 0x4106a9)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/querier.go:520 +0x236 fp=0xc4bdece830 sp=0xc4bdece720 pc=0x14ad456
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*populatedChunkSeries).Next(0xc43a115050, 0xc431461190)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/querier.go:567 +0x3bb fp=0xc4bdece910 sp=0xc4bdece830 pc=0x14adc8b
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*blockSeriesSet).Next(0xc4ad3d2dc0, 0xc4bdece9f0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/querier.go:628 +0x48 fp=0xc4bdece9b8 sp=0xc4bdece910 pc=0x14ade58
github.com/prometheus/prometheus/storage/tsdb.seriesSet.Next(0x1c47a00, 0xc4ad3d2dc0, 0xc431461190)
        /go/src/github.com/prometheus/prometheus/storage/tsdb/tsdb.go:211 +0x31 fp=0xc4bdece9d8 sp=0xc4bdece9b8 pc=0x14c1221
github.com/prometheus/prometheus/storage/tsdb.(*seriesSet).Next(0xc4bcbcc280, 0x1c41340)
        <autogenerated>:1 +0x45 fp=0xc4bdecea00 sp=0xc4bdece9d8 pc=0x14c29a5
github.com/prometheus/prometheus/promql.expandSeriesSet(0x1c4f240, 0xc4bcbcc280, 0xc498634cf8, 0x1, 0x1, 0x1c4f240, 0xc4bcbcc280)
        /go/src/github.com/prometheus/prometheus/promql/engine.go:567 +0x56 fp=0xc4bdecea78 sp=0xc4bdecea00 pc=0x14c7876
github.com/prometheus/prometheus/promql.(*Engine).populateIterators.func2(0x1c36f80, 0xc42043d490, 0xc4ad3d2d80, 0x3, 0x4, 0xc42002e000)
        /go/src/github.com/prometheus/prometheus/promql/engine.go:532 +0x7be fp=0xc4bdeceb88 sp=0xc4bdecea78 pc=0x14e7c3e
github.com/prometheus/prometheus/promql.inspector.Visit(0xc4bcbca300, 0x1c36f80, 0xc42043d490, 0xc4ad3d2d80, 0x3, 0x4, 0xc4bcbc83e0, 0x20)
        /go/src/github.com/prometheus/prometheus/promql/ast.go:307 +0x5c fp=0xc4bdecebc8 sp=0xc4bdeceb88 pc=0x14c3c4c
github.com/prometheus/prometheus/promql.Walk(0x1c3a8c0, 0xc4bcbca300, 0x1c36f80, 0xc42043d490, 0xc4ad3d2d80, 0x3, 0x4)
        /go/src/github.com/prometheus/prometheus/promql/ast.go:255 +0x8a fp=0xc4bdecec58 sp=0xc4bdecebc8 pc=0x14c300a
github.com/prometheus/prometheus/promql.Walk(0x1c3a8c0, 0xc4bcbca300, 0x1c3a880, 0xc4bcbc8400, 0xc4ad3d2d80, 0x3, 0x4)
        /go/src/github.com/prometheus/prometheus/promql/ast.go:276 +0x268 fp=0xc4bdecece8 sp=0xc4bdecec58 pc=0x14c31e8
github.com/prometheus/prometheus/promql.Walk(0x1c3a8c0, 0xc4bcbca300, 0x1c36f40, 0xc4bcbc8260, 0xc4bcbc83e0, 0x2, 0x2)
        /go/src/github.com/prometheus/prometheus/promql/ast.go:286 +0x785 fp=0xc4bdeced78 sp=0xc4bdecece8 pc=0x14c3705
github.com/prometheus/prometheus/promql.Walk(0x1c3a8c0, 0xc4bcbca300, 0x1c36f20, 0xc4ad3d2cc0, 0xc4bcbcc250, 0x1, 0x1)
        /go/src/github.com/prometheus/prometheus/promql/ast.go:282 +0x9ac fp=0xc4bdecee08 sp=0xc4bdeced78 pc=0x14c392c
github.com/prometheus/prometheus/promql.Inspect(0x1c36f20, 0xc4ad3d2cc0, 0xc4bcbca300)
        /go/src/github.com/prometheus/prometheus/promql/ast.go:317 +0x5c fp=0xc4bdecee50 sp=0xc4bdecee08 pc=0x14c3cfc
github.com/prometheus/prometheus/promql.(*Engine).populateIterators(0xc42003ba40, 0x1c50e00, 0xc4bcb50fc0, 0x7f5704ca12d8, 0xc420585100, 0xc4bcb4cf50, 0xc400000073, 0x14e35ae, 0xc4822aa078, 0x1ade7cf)
        /go/src/github.com/prometheus/prometheus/promql/engine.go:499 +0x3a1 fp=0xc4bdecef48 sp=0xc4bdecee50 pc=0x14c75b1
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc42003ba40, 0x1c50e00, 0xc4bcb50fc0, 0xc4bcb4cfa0, 0xc4bcb4cf50, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/promql/engine.go:346 +0xf1 fp=0xc4bdecf4f0 sp=0xc4bdecef48 pc=0x14c5b61
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc42003ba40, 0x1c50e00, 0xc4bcb50fc0, 0xc4bcb4cfa0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/promql/engine.go:327 +0x35f fp=0xc4bdecf560 sp=0xc4bdecf4f0 pc=0x14c585f
github.com/prometheus/prometheus/promql.(*query).Exec(0xc4bcb4cfa0, 0x1c50d80, 0xc4ad3d2c40, 0xc4976446c0)
        /go/src/github.com/prometheus/prometheus/promql/engine.go:130 +0xb8 fp=0xc4bdecf5f8 sp=0xc4bdecf560 pc=0x14c3fd8
github.com/prometheus/prometheus/web/api/v1.(*API).queryRange(0xc4202b9420, 0xc434f67d00, 0x0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/web/api/v1/api.go:310 +0x7f3 fp=0xc4bdecf728 sp=0xc4bdecf5f8 pc=0x15ce843
github.com/prometheus/prometheus/web/api/v1.(*API).(github.com/prometheus/prometheus/web/api/v1.queryRange)-fm(0xc434f67d00, 0xc4bcbc8240, 0xc4bcb4cf00, 0x1)
        /go/src/github.com/prometheus/prometheus/web/api/v1/api.go:176 +0x34 fp=0xc4bdecf760 sp=0xc4bdecf728 pc=0x15d51a4
github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x1c45e00, 0xc4bcbc8240, 0xc434f67d00)
        /go/src/github.com/prometheus/prometheus/web/api/v1/api.go:159 +0x55 fp=0xc4bdecf7a0 sp=0xc4bdecf760 pc=0x15d4b45
net/http.HandlerFunc.ServeHTTP(0xc42052c1c0, 0x1c45e00, 0xc4bcbc8240, 0xc434f67d00)
        /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4bdecf7c8 sp=0xc4bdecf7a0 pc=0x68e9c4
github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x1c3b940, 0xc42052c1c0, 0x1c47640, 0xc4bcb4cf00, 0xc434f67d00)
        /go/src/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7c fp=0xc4bdecf800 sp=0xc4bdecf7c8 pc=0x88942c
github.com/prometheus/prometheus/util/httputil.(*CompressionHandler).ServeHTTP(0xc42052c1d0, 0x1c47640, 0xc4bcb4cf00, 0xc434f67d00)
        <autogenerated>:1 +0x63 fp=0xc4bdecf838 sp=0xc4bdecf800 pc=0x889633
net/http.(Handler).ServeHTTP-fm(0x1c47640, 0xc4bcb4cf00, 0xc434f67d00)
        /usr/local/go/src/net/http/h2_bundle.go:5475 +0x4d fp=0xc4bdecf868 sp=0xc4bdecf838 pc=0x6af2bd
github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x1c47580, 0xc4bcbc8220, 0xc434f67d00)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/http.go:309 +0x229 fp=0xc4bdecf958 sp=0xc4bdecf868 pc=0x775619
github.com/prometheus/prometheus/web.(*Handler).testReady.func1(0x1c47580, 0xc4bcbc8220, 0xc434f67d00)
        /go/src/github.com/prometheus/prometheus/web/web.go:357 +0x55 fp=0xc4bdecf9b8 sp=0xc4bdecf958 pc=0x15fa8d5
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0x1c47580, 0xc4bcbc8220, 0xc434f67b00, 0x0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:50 +0x26a fp=0xc4bdecfa70 sp=0xc4bdecf9b8 pc=0x15cba1a
github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc42020cdc0, 0x1c47580, 0xc4bcbc8220, 0xc434f67b00)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0x6d1 fp=0xc4bdecfb50 sp=0xc4bdecfa70 pc=0x15c7b51
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc420352920, 0x1c47580, 0xc4bcbc8220, 0xc434f67b00)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:88 +0x4c fp=0xc4bdecfb80 sp=0xc4bdecfb50 pc=0x15cb67c
net/http.StripPrefix.func1(0x1c47580, 0xc4bcbc8220, 0xc434f67900)
        /usr/local/go/src/net/http/server.go:1986 +0x19a fp=0xc4bdecfbf0 sp=0xc4bdecfb80 pc=0x6a8f2a
net/http.HandlerFunc.ServeHTTP(0xc420cea240, 0x1c47580, 0xc4bcbc8220, 0xc434f67900)
        /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4bdecfc18 sp=0xc4bdecfbf0 pc=0x68e9c4
net/http.(*ServeMux).ServeHTTP(0xc4203f0ff0, 0x1c47580, 0xc4bcbc8220, 0xc434f67900)
        /usr/local/go/src/net/http/server.go:2337 +0x130 fp=0xc4bdecfc58 sp=0xc4bdecfc18 pc=0x690730
github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2(0x1c4eec0, 0xc43e99fb20, 0xc434f67700)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x3f3 fp=0xc4bdecfd30 sp=0xc4bdecfc58 pc=0x15c5ed3
net/http.HandlerFunc.ServeHTTP(0xc420cea2d0, 0x1c4eec0, 0xc43e99fb20, 0xc434f67700)
        /usr/local/go/src/net/http/server.go:1947 +0x44 fp=0xc4bdecfd58 sp=0xc4bdecfd30 pc=0x68e9c4
net/http.serverHandler.ServeHTTP(0xc420cfa000, 0x1c4eec0, 0xc43e99fb20, 0xc434f67700)
        /usr/local/go/src/net/http/server.go:2694 +0xbc fp=0xc4bdecfd88 sp=0xc4bdecfd58 pc=0x6917dc
net/http.(*conn).serve(0xc46a0520a0, 0x1c50d80, 0xc441f44000)
        /usr/local/go/src/net/http/server.go:1830 +0x651 fp=0xc4bdecffc8 sp=0xc4bdecfd88 pc=0x68d9e1
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4bdecffd0 sp=0xc4bdecffc8 pc=0x45aa01
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2795 +0x27b

goroutine 1 [chan receive, 37 minutes]:
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run(0xc4204cbc88, 0xc420408060, 0x8)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:43 +0xec
main.main()
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:581 +0x5237

goroutine 6 [syscall, 37 minutes]:
os/signal.signal_recv(0x0)
        /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 18 [chan receive]:
github.com/prometheus/prometheus/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x2883600)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:879 +0x8b
created by github.com/prometheus/prometheus/vendor/github.com/golang/glog.init.0
        /go/src/github.com/prometheus/prometheus/vendor/github.com/golang/glog/glog.go:410 +0x203

goroutine 267 [chan receive]:
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420352820, 0x1b6e040, 0xc4bcb90320, 0x1c50e40, 0xc420348630)
        <autogenerated>:1 +0x5b
net/http.(*Server).Serve(0xc420cfa000, 0x1c4f480, 0xc420352820, 0x0, 0x0)
        /usr/local/go/src/net/http/server.go:2770 +0x1a5
github.com/prometheus/prometheus/web.(*Handler).Run.func5(0xc420cfa000, 0x1c4f480, 0xc420352820, 0xc42023f600)
        /go/src/github.com/prometheus/prometheus/web/web.go:455 +0x43
created by github.com/prometheus/prometheus/web.(*Handler).Run
        /go/src/github.com/prometheus/prometheus/web/web.go:454 +0xced

goroutine 224 [select]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*SegmentWAL).run(0xc420442000, 0x2540be400)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:704 +0x36e
created by github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.OpenSegmentWAL
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/wal.go:244 +0x776

goroutine 181 [select, 37 minutes, locked to thread]:
runtime.gopark(0x1b6e6d0, 0x0, 0x1ade841, 0x6, 0x18, 0x1)
        /usr/local/go/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc420473750, 0xc42023c2a0)
        /usr/local/go/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1

goroutine 69 [select, 37 minutes]:
main.main.func6(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:372 +0x121
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc42020ca40, 0xc4200834b0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 70 [chan receive, 37 minutes]:
github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc4202b8fc0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:93 +0x50
main.main.func8(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:393 +0x40
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc420458640, 0xc420458680)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 71 [chan receive, 37 minutes]:
github.com/prometheus/prometheus/discovery.(*Manager).Run(0xc4202b9030, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:93 +0x50
main.main.func10(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:406 +0x40
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc4204586c0, 0xc4204586e0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 72 [select, 37 minutes]:
github.com/prometheus/prometheus/scrape.(*Manager).Run(0xc4200b5c70, 0xc42027e2a0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/scrape/manager.go:60 +0xfc
main.main.func12(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:425 +0x7a
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc4203f07e0, 0xc420458720)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 73 [select, 37 minutes]:
main.main.func14(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:448 +0x18a
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc420080120, 0xc420168000)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 74 [chan receive, 37 minutes]:
main.main.func16(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:492 +0x325
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc420080180, 0xc420408020)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 75 [chan receive, 37 minutes]:
main.main.func18(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:519 +0x41d
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc42020cac0, 0xc4203f0810)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 76 [select, 37 minutes]:
github.com/prometheus/prometheus/web.(*Handler).Run(0xc42023f600, 0x1c50d80, 0xc420585140, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/web/web.go:470 +0xe6b
main.main.func20(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:533 +0x40
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc420352060, 0xc420408050)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 77 [chan receive, 37 minutes]:
main.main.func22(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:552 +0x4e
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc420352080, 0xc4203520c0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 78 [select]:
github.com/prometheus/prometheus/notifier.(*Manager).Run(0xc420016c00, 0xc42027e300)
        /go/src/github.com/prometheus/prometheus/notifier/notifier.go:303 +0x133
main.main.func24(0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:572 +0x7a
github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run.func1(0xc4201621e0, 0xc4203f0840, 0xc420408060)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:38 +0x27
created by github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group.(*Group).Run
        /go/src/github.com/prometheus/prometheus/vendor/github.com/oklog/oklog/pkg/group/group.go:37 +0xa8

goroutine 193 [chan receive (nil chan), 37 minutes]:
github.com/prometheus/prometheus/prompb.RegisterAdminHandlerFromEndpoint.func1.1(0x7f5704c54058, 0xc420040018, 0xc4200c61a0, 0x1ae4f89, 0xc)
        /go/src/github.com/prometheus/prometheus/prompb/rpc.pb.gw.go:85 +0x4c
created by github.com/prometheus/prometheus/prompb.RegisterAdminHandlerFromEndpoint.func1
        /go/src/github.com/prometheus/prometheus/prompb/rpc.pb.gw.go:84 +0x19b

goroutine 80 [select, 37 minutes]:
github.com/prometheus/prometheus/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4200c6340)
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/clientconn.go:908 +0x1c0
github.com/prometheus/prometheus/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn.func1(0xc4200c6340)
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/clientconn.go:637 +0x1af
created by github.com/prometheus/prometheus/vendor/google.golang.org/grpc.(*ClientConn).resetAddrConn
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/clientconn.go:628 +0x6d8

goroutine 401 [select]:
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc43e35c300, 0x12a05f200, 0x12a05f200, 0x0)
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:714 +0x93d
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:303 +0x41f

goroutine 29 [IO wait, 37 minutes]:
internal/poll.runtime_pollWait(0x7f5704c98e30, 0x72, 0xc420997b88)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4202d8418, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc4202d8418, 0xc4209fc000, 0x8000, 0x8000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc4202d8400, 0xc4209fc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc4202d8400, 0xc4209fc000, 0x8000, 0x8000, 0x60, 0x1a53200, 0xc420048a01)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc420174048, 0xc4209fc000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
bufio.(*Reader).Read(0xc4203326c0, 0xc420a0c038, 0x9, 0x9, 0xc420062fa0, 0x2, 0x2)
        /usr/local/go/src/bufio/bufio.go:216 +0x238
io.ReadAtLeast(0x1c36960, 0xc4203326c0, 0xc420a0c038, 0x9, 0x9, 0x9, 0x2, 0xc420062f20, 0xc420062dd8)
        /usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x1c36960, 0xc4203326c0, 0xc420a0c038, 0x9, 0x9, 0x403f2c, 0xc420098d80, 0x4)
        /usr/local/go/src/io/io.go:327 +0x58
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.readFrameHeader(0xc420a0c038, 0x9, 0x9, 0x1c36960, 0xc4203326c0, 0x0, 0xc400000000, 0x0, 0x2)
        /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420a0c000, 0x0, 0x20002, 0xc4200ac360, 0xc420062f80)
        /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc42016f7d0, 0xc420062fa0, 0x0, 0xc4200ac360, 0x53d70d)
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http_util.go:608 +0x2f
github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420198d80)
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http2_client.go:1080 +0x47
created by github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.newHTTP2Client
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http2_client.go:267 +0xb6c

goroutine 30 [select, 37 minutes]:
github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc420198d80)
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http2_client.go:1168 +0x122
created by github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport.newHTTP2Client
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/transport/http2_client.go:297 +0xca2

goroutine 268 [chan receive, 37 minutes]:
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:184
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*muxListener).Accept(0xc420352300, 0x1b6a478, 0xc42042c900, 0x1c4f480, 0xc420352300)
        <autogenerated>:1 +0x5b
github.com/prometheus/prometheus/vendor/google.golang.org/grpc.(*Server).Serve(0xc42042c900, 0x1c4f480, 0xc420352300, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/google.golang.org/grpc/server.go:432 +0x16c
github.com/prometheus/prometheus/web.(*Handler).Run.func6(0xc42042c900, 0x1c4f480, 0xc420352300, 0xc42023f600)
        /go/src/github.com/prometheus/prometheus/web/web.go:460 +0x43
created by github.com/prometheus/prometheus/web.(*Handler).Run
        /go/src/github.com/prometheus/prometheus/web/web.go:459 +0xd39

goroutine 269 [IO wait]:
internal/poll.runtime_pollWait(0x7f5704c98f00, 0x72, 0x0)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42007a298, 0x72, 0xc420042000, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc42007a298, 0xffffffffffffff00, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc42007a280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:372 +0x1a8
net.(*netFD).accept(0xc42007a280, 0x1c59ea0, 0xc42023c538, 0x1adc177)
        /usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc4201f8008, 0xc42023c4e0, 0xc42099be98, 0x1)
        /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc4201f8008, 0xc42099be98, 0x1955540, 0x1955540, 0xc4bcb616e0)
        /usr/local/go/src/net/tcpsock.go:259 +0x49
github.com/prometheus/prometheus/vendor/golang.org/x/net/netutil.(*limitListener).Accept(0xc420352180, 0x4346f4, 0xc42099bee8, 0x4571a0, 0xc42099bf28)
        /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/netutil/listen.go:30 +0x53
github.com/prometheus/prometheus/vendor/github.com/mwitkow/go-conntrack.(*connTrackListener).Accept(0xc4203522a0, 0x1b678a0, 0xc42020cbc0, 0x1c5d9e0, 0xc4bcb61770)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/mwitkow/go-conntrack/listener_wrapper.go:86 +0x37
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc42020cbc0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88
github.com/prometheus/prometheus/web.(*Handler).Run.func7(0xc420a3a060, 0x1c45ec0, 0xc42020cbc0)
        /go/src/github.com/prometheus/prometheus/web/web.go:467 +0x31
created by github.com/prometheus/prometheus/web.(*Handler).Run
        /go/src/github.com/prometheus/prometheus/web/web.go:466 +0xd8e

goroutine 270 [IO wait, 37 minutes]:
internal/poll.runtime_pollWait(0x7f5704c98d60, 0x72, 0xc420d09940)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc420416798, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc420416798, 0xc420d16000, 0x9, 0x9)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc420416780, 0xc420d16038, 0x9, 0x9, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc420416780, 0xc420d16038, 0x9, 0x9, 0x4, 0x0, 0x0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc420a403a0, 0xc420d16038, 0x9, 0x9, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*bufferedReader).Read(0xc4201d7518, 0xc420d16038, 0x9, 0x9, 0x100000000000000, 0x0, 0x10)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/buffer.go:42 +0x120
io.ReadAtLeast(0x1c37180, 0xc4201d7518, 0xc420d16038, 0x9, 0x9, 0x9, 0x410ef8, 0x10, 0x1964300)
        /usr/local/go/src/io/io.go:309 +0x86
io.ReadFull(0x1c37180, 0xc4201d7518, 0xc420d16038, 0x9, 0x9, 0xd7b4f57b212aa901, 0xefff100000004, 0x2)
        /usr/local/go/src/io/io.go:327 +0x58
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.readFrameHeader(0xc420d16038, 0x9, 0x9, 0x1c37180, 0xc4201d7518, 0x0, 0x0, 0xc420040200, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
github.com/prometheus/prometheus/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420d16000, 0x1c3eb80, 0xc420040200, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.matchHTTP2Field(0x1c37180, 0xc4201d7518, 0x1ae59d9, 0xc, 0x1aeaa9a, 0x10, 0x7f57049d4308)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/matchers.go:145 +0x140
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.HTTP2HeaderField.func1(0x1c37180, 0xc4201d7518, 0xc420cea390)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/matchers.go:111 +0x59
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*cMux).serve(0xc42020cbc0, 0x1c5d9e0, 0xc420cea390, 0xc42023c540, 0xc420560380)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:143 +0x1f3
created by github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*cMux).Serve
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:133 +0x15d

goroutine 300 [IO wait]:
internal/poll.runtime_pollWait(0x7f5704c987b0, 0x72, 0xc420696960)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc428c20f18, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc428c20f18, 0xc43e30c000, 0x1000, 0x1000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc428c20f00, 0xc43e30c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc428c20f00, 0xc43e30c000, 0x1000, 0x1000, 0xc498634cf0, 0xc425c9c000, 0x4)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc428c2a080, 0xc43e30c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
net/http.(*persistConn).Read(0xc425c9c000, 0xc43e30c000, 0x1000, 0x1000, 0xc420696b98, 0x403855, 0xc4200accc0)
        /usr/local/go/src/net/http/transport.go:1453 +0x136
bufio.(*Reader).fill(0xc428c0e6c0)
        /usr/local/go/src/bufio/bufio.go:100 +0x11e
bufio.(*Reader).Peek(0xc428c0e6c0, 0x1, 0x0, 0x0, 0x0, 0xc497641380, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0x3a
net/http.(*persistConn).readLoop(0xc425c9c000)
        /usr/local/go/src/net/http/transport.go:1601 +0x185
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1237 +0x95a

goroutine 283 [select]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run(0xc420468180)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:265 +0x25c
created by github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.Open
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:243 +0x5ba

goroutine 368 [select]:
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc43e35c080, 0x12a05f200, 0x12a05f200, 0x0)
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:714 +0x93d
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:303 +0x41f

goroutine 367 [select]:
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc452e49e00, 0x2540be400, 0x2540be400, 0x0)
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:714 +0x93d
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:303 +0x41f

goroutine 420 [IO wait]:
internal/poll.runtime_pollWait(0x7f5704c98880, 0x72, 0xc451262960)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42a189418, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc42a189418, 0xc453ed8000, 0x1000, 0x1000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42a189400, 0xc453ed8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc42a189400, 0xc453ed8000, 0x1000, 0x1000, 0xc44e68e4f8, 0xc440666240, 0x4)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc453ea1980, 0xc453ed8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
net/http.(*persistConn).Read(0xc440666240, 0xc453ed8000, 0x1000, 0x1000, 0xc451262b98, 0x403855, 0xc420a3b3e0)
        /usr/local/go/src/net/http/transport.go:1453 +0x136
bufio.(*Reader).fill(0xc42a19baa0)
        /usr/local/go/src/bufio/bufio.go:100 +0x11e
bufio.(*Reader).Peek(0xc42a19baa0, 0x1, 0x0, 0x0, 0x0, 0xc44e6a4f60, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0x3a
net/http.(*persistConn).readLoop(0xc440666240)
        /usr/local/go/src/net/http/transport.go:1601 +0x185
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1237 +0x95a

goroutine 366 [select]:
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc452e49b80, 0x12a05f200, 0x12a05f200, 0x0)
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:714 +0x93d
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:303 +0x41f

goroutine 365 [select]:
github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc452e49900, 0x12a05f200, 0x12a05f200, 0x0)
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:714 +0x93d
created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync
        /go/src/github.com/prometheus/prometheus/scrape/scrape.go:303 +0x41f

goroutine 286 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).runUpdater(0xc4202b8fc0, 0x1c50d80, 0xc4308dca40)
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:155 +0x13a
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:128 +0x1cc

goroutine 438 [IO wait]:
internal/poll.runtime_pollWait(0x7f5704c98a20, 0x72, 0xc425e22960)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc45124a218, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc45124a218, 0xc435059000, 0x1000, 0x1000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc45124a200, 0xc435059000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc45124a200, 0xc435059000, 0x1000, 0x1000, 0xc44e68e4d0, 0xc440666120, 0x4)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc4526f1ab8, 0xc435059000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
net/http.(*persistConn).Read(0xc440666120, 0xc435059000, 0x1000, 0x1000, 0xc425e22b98, 0x403855, 0xc420a3aba0)
        /usr/local/go/src/net/http/transport.go:1453 +0x136
bufio.(*Reader).fill(0xc4200ab9e0)
        /usr/local/go/src/bufio/bufio.go:100 +0x11e
bufio.(*Reader).Peek(0xc4200ab9e0, 0x1, 0x0, 0x0, 0x0, 0xc4bcb5e1e0, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0x3a
net/http.(*persistConn).readLoop(0xc440666120)
        /usr/local/go/src/net/http/transport.go:1601 +0x185
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1237 +0x95a

goroutine 337 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).runUpdater(0xc4202b8fc0, 0x1c50d80, 0xc4308dca80)
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:155 +0x13a
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:128 +0x1cc

goroutine 440 [IO wait]:
internal/poll.runtime_pollWait(0x7f5704c98af0, 0x72, 0xc420694960)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc45126a798, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc45126a798, 0xc45124d000, 0x1000, 0x1000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc45126a780, 0xc45124d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc45126a780, 0xc45124d000, 0x1000, 0x1000, 0xc4b35a7228, 0xc43b1bb9e0, 0x4)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc4526f1ae0, 0xc45124d000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
net/http.(*persistConn).Read(0xc43b1bb9e0, 0xc45124d000, 0x1000, 0x1000, 0xc420694b98, 0x403855, 0xc44197ef60)
        /usr/local/go/src/net/http/transport.go:1453 +0x136
bufio.(*Reader).fill(0xc4200abaa0)
        /usr/local/go/src/bufio/bufio.go:100 +0x11e
bufio.(*Reader).Peek(0xc4200abaa0, 0x1, 0x0, 0x0, 0x0, 0xc44e6a4e40, 0x0)
        /usr/local/go/src/bufio/bufio.go:132 +0x3a
net/http.(*persistConn).readLoop(0xc43b1bb9e0)
        /usr/local/go/src/net/http/transport.go:1601 +0x185
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1237 +0x95a

goroutine 441 [select]:
net/http.(*persistConn).writeLoop(0xc43b1bb9e0)
        /usr/local/go/src/net/http/transport.go:1822 +0x14b
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1238 +0x97f

goroutine 340 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).runUpdater(0xc4202b8fc0, 0x1c50d80, 0xc4308dcac0)
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:155 +0x13a
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:128 +0x1cc

goroutine 343 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).runUpdater(0xc4202b8fc0, 0x1c50d80, 0xc4308dcb00)
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:155 +0x13a
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:128 +0x1cc

goroutine 346 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).runUpdater(0xc4202b8fc0, 0x1c50d80, 0xc4308dcb40)
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:155 +0x13a
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:128 +0x1cc

goroutine 349 [select]:
github.com/prometheus/prometheus/discovery.(*Manager).runUpdater(0xc4202b9030, 0x1c50d80, 0xc4308dcbc0)
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:155 +0x13a
created by github.com/prometheus/prometheus/discovery.(*Manager).startProvider
        /go/src/github.com/prometheus/prometheus/discovery/manager.go:128 +0x1cc

goroutine 96 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc451b64ea0, 0x1c50dc0, 0xc420040018)
        /go/src/github.com/prometheus/prometheus/rules/manager.go:220 +0x355
github.com/prometheus/prometheus/rules.(*Manager).Update.func1.1(0xc4205855c0, 0xc451b64ea0)
        /go/src/github.com/prometheus/prometheus/rules/manager.go:505 +0x60
created by github.com/prometheus/prometheus/rules.(*Manager).Update.func1
        /go/src/github.com/prometheus/prometheus/rules/manager.go:500 +0x56

goroutine 421 [select]:
net/http.(*persistConn).writeLoop(0xc440666240)
        /usr/local/go/src/net/http/transport.go:1822 +0x14b
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1238 +0x97f

goroutine 201 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc451b64f30, 0x1c50dc0, 0xc420040018)
        /go/src/github.com/prometheus/prometheus/rules/manager.go:220 +0x355
github.com/prometheus/prometheus/rules.(*Manager).Update.func1.1(0xc4205855c0, 0xc451b64f30)
        /go/src/github.com/prometheus/prometheus/rules/manager.go:505 +0x60
created by github.com/prometheus/prometheus/rules.(*Manager).Update.func1
        /go/src/github.com/prometheus/prometheus/rules/manager.go:500 +0x56

goroutine 439 [select]:
net/http.(*persistConn).writeLoop(0xc440666120)
        /usr/local/go/src/net/http/transport.go:1822 +0x14b
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1238 +0x97f

goroutine 383 [select]:
github.com/prometheus/prometheus/rules.(*Group).run(0xc451b64fc0, 0x1c50dc0, 0xc420040018)
        /go/src/github.com/prometheus/prometheus/rules/manager.go:220 +0x355
github.com/prometheus/prometheus/rules.(*Manager).Update.func1.1(0xc4205855c0, 0xc451b64fc0)
        /go/src/github.com/prometheus/prometheus/rules/manager.go:505 +0x60
created by github.com/prometheus/prometheus/rules.(*Manager).Update.func1
        /go/src/github.com/prometheus/prometheus/rules/manager.go:500 +0x56

goroutine 307 [IO wait]:
internal/poll.runtime_pollWait(0x7f5704c98950, 0x72, 0xc425ca5aa8)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc425cc2c18, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc425cc2c18, 0xc453f07000, 0x1000, 0x1000)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc425cc2c00, 0xc453f07000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc425cc2c00, 0xc453f07000, 0x1000, 0x1000, 0xc425ca5bd0, 0x18, 0xc425ca5bc0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc425cf6040, 0xc453f07000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4413e2000, 0xc453f07000, 0x1000, 0x1000, 0x2883180, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x119
net/http.(*connReader).Read(0xc441b56030, 0xc453f07000, 0x1000, 0x1000, 0x15c43ba, 0xc425cf4810, 0xbeb5187b412f3a86)
        /usr/local/go/src/net/http/server.go:764 +0xf8
bufio.(*Reader).fill(0xc441b60000)
        /usr/local/go/src/bufio/bufio.go:100 +0x11e
bufio.(*Reader).Peek(0xc441b60000, 0x4, 0x25239290ac3, 0x2883180, 0x0, 0x0, 0x2883180)
        /usr/local/go/src/bufio/bufio.go:132 +0x3a
net/http.(*conn).serve(0xc441b52000, 0x1c50d80, 0xc441b5a000)
        /usr/local/go/src/net/http/server.go:1855 +0x7c3
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2795 +0x27b

goroutine 301 [select]:
net/http.(*persistConn).writeLoop(0xc425c9c000)
        /usr/local/go/src/net/http/transport.go:1822 +0x14b
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1238 +0x97f

goroutine 55396 [IO wait]:
internal/poll.runtime_pollWait(0x7f5704c98540, 0x72, 0xc441c31d60)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc45123c018, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc45123c018, 0xc44c1b2100, 0x1, 0x1)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc45123c000, 0xc44c1b2101, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc45123c000, 0xc44c1b2101, 0x1, 0x1, 0x183a420, 0xc441c31e98, 0x42a864)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc430540700, 0xc44c1b2101, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc46a052000, 0xc44c1b2101, 0x1, 0x1, 0xc43cc91a90, 0x433508, 0x1b6e540)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x119
net/http.(*connReader).backgroundRead(0xc44c1b20f0)
        /usr/local/go/src/net/http/server.go:668 +0x5a
created by net/http.(*connReader).startBackgroundRead
        /usr/local/go/src/net/http/server.go:664 +0xce

goroutine 55637 [semacquire]:
sync.runtime_Semacquire(0xc44e69f914)
        /usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc44e69f908)
        /usr/local/go/src/sync/waitgroup.go:129 +0x72
github.com/prometheus/prometheus/vendor/golang.org/x/sync/errgroup.(*Group).Wait(0xc44e69f900, 0xc44e69fa00, 0xc4bcbc7150)
        /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/sync/errgroup/errgroup.go:41 +0x31
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).Delete(0xc420468180, 0x80003883122cdf10, 0x16348903768, 0xc4bcbc7150, 0x1, 0x1, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:728 +0x2e8
github.com/prometheus/prometheus/web/api/v1.(*API).deleteSeries(0xc4202b9420, 0xc44e6a7300, 0x8000103, 0x0, 0xffffffffffffffff)
        /go/src/github.com/prometheus/prometheus/web/api/v1/api.go:648 +0x585
github.com/prometheus/prometheus/web/api/v1.(*API).(github.com/prometheus/prometheus/web/api/v1.deleteSeries)-fm(0xc44e6a7300, 0xc4bcbacca0, 0xc497627f40, 0x1)
        /go/src/github.com/prometheus/prometheus/web/api/v1/api.go:192 +0x34
github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x1c45e00, 0xc4bcbacca0, 0xc44e6a7300)
        /go/src/github.com/prometheus/prometheus/web/api/v1/api.go:159 +0x55
net/http.HandlerFunc.ServeHTTP(0xc42031c2f0, 0x1c45e00, 0xc4bcbacca0, 0xc44e6a7300)
        /usr/local/go/src/net/http/server.go:1947 +0x44
github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x1c3b940, 0xc42031c2f0, 0x1c47640, 0xc497627f40, 0xc44e6a7300)
        /go/src/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7c
net/http.(Handler).ServeHTTP-fm(0x1c47640, 0xc497627f40, 0xc44e6a7300)
        /usr/local/go/src/net/http/h2_bundle.go:5475 +0x4d
github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus.InstrumentHandlerFuncWithOpts.func1(0x1c47580, 0xc4bcbacc80, 0xc44e6a7300)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/client_golang/prometheus/http.go:309 +0x229
github.com/prometheus/prometheus/web.(*Handler).testReady.func1(0x1c47580, 0xc4bcbacc80, 0xc44e6a7300)
        /go/src/github.com/prometheus/prometheus/web/web.go:357 +0x55
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).handle.func1(0x1c47580, 0xc4bcbacc80, 0xc44e6a7200, 0x0, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:50 +0x26a
github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc42020cdc0, 0x1c47580, 0xc4bcbacc80, 0xc44e6a7200)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/julienschmidt/httprouter/router.go:299 +0x6d1
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route.(*Router).ServeHTTP(0xc420352920, 0x1c47580, 0xc4bcbacc80, 0xc44e6a7200)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/route/route.go:88 +0x4c
net/http.StripPrefix.func1(0x1c47580, 0xc4bcbacc80, 0xc44e6a7100)
        /usr/local/go/src/net/http/server.go:1986 +0x19a
net/http.HandlerFunc.ServeHTTP(0xc420cea240, 0x1c47580, 0xc4bcbacc80, 0xc44e6a7100)
        /usr/local/go/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0xc4203f0ff0, 0x1c47580, 0xc4bcbacc80, 0xc44e6a7100)
        /usr/local/go/src/net/http/server.go:2337 +0x130
github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2(0x1c4eec0, 0xc427211500, 0xc44e6a7000)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x3f3
net/http.HandlerFunc.ServeHTTP(0xc420cea2d0, 0x1c4eec0, 0xc427211500, 0xc44e6a7000)
        /usr/local/go/src/net/http/server.go:1947 +0x44
net/http.serverHandler.ServeHTTP(0xc420cfa000, 0x1c4eec0, 0xc427211500, 0xc44e6a7000)
        /usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc4bcb90320, 0x1c50d80, 0xc44e69f7c0)
        /usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2795 +0x27b

goroutine 55638 [IO wait]:
internal/poll.runtime_pollWait(0x7f5704c98200, 0x72, 0xc487fead60)
        /usr/local/go/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc44e6a8918, 0x72, 0xffffffffffffff00, 0x1c3be60, 0x27b7638)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc44e6a8918, 0xc4bcba1a00, 0x1, 0x1)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc44e6a8900, 0xc4bcba1a51, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc44e6a8900, 0xc4bcba1a51, 0x1, 0x1, 0x0, 0x2, 0xc4203f09c0)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc438552f18, 0xc4bcba1a51, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:176 +0x6a
github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc4bcb90280, 0xc4bcba1a51, 0x1, 0x1, 0x0, 0x3000100, 0x0)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x119
net/http.(*connReader).backgroundRead(0xc4bcba1a40)
        /usr/local/go/src/net/http/server.go:668 +0x5a
created by net/http.(*connReader).startBackgroundRead
        /usr/local/go/src/net/http/server.go:664 +0xce

goroutine 55644 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.memTombstones.add(0xc420bedb60, 0x1bbd1, 0x1634840ba8b, 0x16348903768)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/tombstones.go:181 +0x105
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*Head).Delete(0xc4204421e0, 0x80003883122cdf10, 0x16348903768, 0xc4bcbc7150, 0x1, 0x1, 0x14d8ebb, 0xc439a38f90)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/head.go:602 +0x3bf
github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).Delete.func2(0x8, 0x1b6e848)
        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:726 +0x5e
github.com/prometheus/prometheus/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc44e69f900, 0xc44e69fa00)
        /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57
created by github.com/prometheus/prometheus/vendor/golang.org/x/sync/errgroup.(*Group).Go
        /go/src/github.com/prometheus/prometheus/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66

@brian-brazil brian-brazil added kind/bug component/local storage labels May 11, 2018

@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented May 11, 2018

I run cronjob to delete metric data every 5min

As an aside, using delete this often is really not recommended.
@games130

This comment has been minimized.

Sign in to view
Copy link
Author

games130 commented May 11, 2018

Any guideline on the timing as not to incur this error? every 30min?
@gouthamve

This comment has been minimized.

Sign in to view
Copy link
Member

gouthamve commented May 11, 2018

This is a bug, and should be fixed. If you are running into this with a cron of 5mins, you'll likely run into this for 30mins too.
@games130

This comment has been minimized.

Sign in to view
Copy link
Author

games130 commented May 14, 2018

ahh... ic ic. okay i will wait for the bug fix. Thank You
@codwu

This comment has been minimized.

Sign in to view
Copy link

codwu commented May 14, 2018

this can be fixed by adding some lines to prometheus/tsdb/querier.go, like

var rm sync.RWMutex
rm.RLock()
s.intervals, err = s.tombstones.Get(s.p.At())
rm.Unlock()
@krasi-georgiev

This comment has been minimized.

Sign in to view
Copy link
Member

krasi-georgiev commented May 14, 2018

@codwu thanks! , can you open a PR and I will review it.

@codwu codwu referenced this issue May 14, 2018

Merged

add rwmutex to prevent concurrent map read when delete series #330

@games130

This comment has been minimized.

Sign in to view
Copy link
Author

games130 commented Jun 5, 2018

When testing the bug fix,
1st time the prometheus stop scrapping the data show error "context deadline exceeded" - prometheus still running
2nd time prometheus stop running all together (docker ps show no prometheus container running). from the log i notice the error "context deadline exceeded" also.

During the first time when prometheus show error "context deadline exceeded", i did a curl and was able to get the metrics from the target. prometheus and target are installed on the same server.

Below is the log:

[root@localhost prometheus]# docker logs  07bdfbc726a7
level=info ts=2018-06-01T01:09:06.361814693Z caller=main.go:218 msg="Starting Prometheus" version="(version=2.2.1, branch=master, revision=7e376dfc8901baaa5efd4d7c2ab92ba46d041d7e)"
level=info ts=2018-06-01T01:09:06.361955698Z caller=main.go:219 build_context="(go=go1.10.2, user=root@94b12c4de194, date=20180529-13:25:24)"
level=info ts=2018-06-01T01:09:06.362008664Z caller=main.go:220 host_details="(Linux 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 localhost.localdomain (none))"
level=info ts=2018-06-01T01:09:06.362048141Z caller=main.go:221 fd_limits="(soft=65536, hard=65536)"
level=info ts=2018-06-01T01:09:06.363654843Z caller=web.go:426 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-01T01:09:06.36367166Z caller=main.go:510 msg="Starting TSDB ..."
level=info ts=2018-06-01T01:09:06.379126252Z caller=main.go:520 msg="TSDB started"
level=info ts=2018-06-01T01:09:06.379266633Z caller=main.go:599 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2018-06-01T01:09:06.415451035Z caller=main.go:496 msg="Server is ready to receive web requests."
level=info ts=2018-06-01T03:00:05.870937298Z caller=compact.go:387 component=tsdb msg="write block" mint=1527811200000 maxt=1527818400000 ulid=01CEWMK8MZ20NGN5MP6WNJ7KQH
level=info ts=2018-06-01T03:00:06.801063258Z caller=head.go:348 component=tsdb msg="head GC completed" duration=381.955392ms
level=info ts=2018-06-01T03:00:07.688961833Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=887.769139ms
level=info ts=2018-06-01T03:00:09.661466342Z caller=compact.go:387 component=tsdb msg="write block" mint=1527811200000 maxt=1527818400000 ulid=01CEWMKGPZWD1ZJDRF2F22SF6M
level=info ts=2018-06-01T05:00:07.914483921Z caller=compact.go:387 component=tsdb msg="write block" mint=1527818400000 maxt=1527825600000 ulid=01CEWVEZWKWWF9R28D06DG40HT
level=info ts=2018-06-01T05:00:09.661417901Z caller=head.go:348 component=tsdb msg="head GC completed" duration=597.566593ms
level=info ts=2018-06-01T05:00:19.297701206Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=9.636165034s
level=info ts=2018-06-01T05:00:21.848272977Z caller=compact.go:387 component=tsdb msg="write block" mint=1527818400000 maxt=1527825600000 ulid=01CEWVFKJCERBDCRMEGS265XNK
level=info ts=2018-06-01T07:00:10.111158519Z caller=compact.go:387 component=tsdb msg="write block" mint=1527825600000 maxt=1527832800000 ulid=01CEX2AQ5JFCYK23KEA4EDXC56
level=info ts=2018-06-01T07:00:12.018088869Z caller=head.go:348 component=tsdb msg="head GC completed" duration=567.560414ms
level=info ts=2018-06-01T07:00:22.715556801Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=10.688783549s
level=info ts=2018-06-01T07:00:26.327877716Z caller=compact.go:387 component=tsdb msg="write block" mint=1527825600000 maxt=1527832800000 ulid=01CEX2BEZVJZ19N4VNX0DXGFQC
level=info ts=2018-06-01T09:00:09.488303211Z caller=compact.go:387 component=tsdb msg="write block" mint=1527832800000 maxt=1527840000000 ulid=01CEX96ECRSMXW2KVYN4DC1399
level=info ts=2018-06-01T09:00:11.782096571Z caller=head.go:348 component=tsdb msg="head GC completed" duration=709.000418ms
level=info ts=2018-06-01T09:00:22.418242284Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=10.63599205s
level=info ts=2018-06-01T09:00:26.21187751Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527811200000 maxt=1527832800000 ulid=01CEX975XVAN4QQAZJYPYK5VNA sources="[01CEWMKGPZWD1ZJDRF2F22SF6M 01CEWVFKJCERBDCRMEGS265XNK 01CEX2BEZVJZ19N4VNX0DXGFQC]"
level=info ts=2018-06-01T09:00:30.837973621Z caller=compact.go:387 component=tsdb msg="write block" mint=1527832800000 maxt=1527840000000 ulid=01CEX97AJ6Y5TPXZZYDWEDDYFA
level=info ts=2018-06-01T11:00:11.607463457Z caller=compact.go:387 component=tsdb msg="write block" mint=1527840000000 maxt=1527847200000 ulid=01CEXG25N4RZZCF7TF0X19KVRJ
level=info ts=2018-06-01T11:00:14.524183163Z caller=head.go:348 component=tsdb msg="head GC completed" duration=684.819994ms
level=info ts=2018-06-01T11:00:34.447210919Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=19.922910992s
level=info ts=2018-06-01T11:00:39.167266304Z caller=compact.go:387 component=tsdb msg="write block" mint=1527840000000 maxt=1527847200000 ulid=01CEXG39P96CNST7FMY03JSX16
level=info ts=2018-06-01T13:00:18.205387363Z caller=compact.go:387 component=tsdb msg="write block" mint=1527847200000 maxt=1527854400000 ulid=01CEXPXWXE6QKGWHZ3ZAC6YS0J
level=info ts=2018-06-01T13:00:21.403542535Z caller=head.go:348 component=tsdb msg="head GC completed" duration=967.513731ms
level=info ts=2018-06-01T13:00:40.339455624Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=18.920598738s
level=info ts=2018-06-01T13:00:46.000472476Z caller=compact.go:387 component=tsdb msg="write block" mint=1527847200000 maxt=1527854400000 ulid=01CEXPZ7FANYNZ1ZK3NFTXAF9X
level=info ts=2018-06-01T15:00:14.642665629Z caller=compact.go:387 component=tsdb msg="write block" mint=1527854400000 maxt=1527861600000 ulid=01CEXXSM4RM9WAVMZ57QG7T297
level=info ts=2018-06-01T15:00:18.664163399Z caller=head.go:348 component=tsdb msg="head GC completed" duration=816.897513ms
level=info ts=2018-06-01T15:00:45.876606797Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=27.212249706s
level=info ts=2018-06-01T15:00:49.628425747Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527832800000 maxt=1527854400000 ulid=01CEXXV204EVMDJBZQ1SZ2BRM4 sources="[01CEX97AJ6Y5TPXZZYDWEDDYFA 01CEXG39P96CNST7FMY03JSX16 01CEXPZ7FANYNZ1ZK3NFTXAF9X]"
level=info ts=2018-06-01T15:00:56.636341919Z caller=compact.go:387 component=tsdb msg="write block" mint=1527854400000 maxt=1527861600000 ulid=01CEXXV8YTY3CC90NC5W46JBX2
level=info ts=2018-06-01T17:02:19.392026531Z caller=compact.go:387 component=tsdb msg="write block" mint=1527861600000 maxt=1527868800000 ulid=01CEY4PZASYQJNVMA1BVCRE8YQ
level=info ts=2018-06-01T17:02:24.791134715Z caller=head.go:348 component=tsdb msg="head GC completed" duration=2.821292944s
level=info ts=2018-06-01T17:03:01.533730208Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=36.708137451s
level=info ts=2018-06-01T17:03:08.830413509Z caller=compact.go:347 component=tsdb msg="compact blocks" count=2 mint=1527811200000 maxt=1527854400000 ulid=01CEY4TYHB1NX270JPQ3M6R74C sources="[01CEX975XVAN4QQAZJYPYK5VNA 01CEXXV204EVMDJBZQ1SZ2BRM4]"
level=info ts=2018-06-01T17:03:35.584934722Z caller=compact.go:387 component=tsdb msg="write block" mint=1527861600000 maxt=1527868800000 ulid=01CEY4VTWPK0TDZJKB5J2YCXXE
level=info ts=2018-06-01T19:00:26.335369183Z caller=compact.go:387 component=tsdb msg="write block" mint=1527868800000 maxt=1527876000000 ulid=01CEYBH2MWYR3QBFKVWM9M9DJG
level=info ts=2018-06-01T19:00:29.301577022Z caller=head.go:348 component=tsdb msg="head GC completed" duration=812.40831ms
level=info ts=2018-06-01T19:00:42.871370157Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=13.543244847s
level=info ts=2018-06-01T19:00:50.51246704Z caller=compact.go:387 component=tsdb msg="write block" mint=1527868800000 maxt=1527876000000 ulid=01CEYBJGXK379EPSWJ3G0FVPCQ
level=info ts=2018-06-01T21:00:20.772970157Z caller=compact.go:387 component=tsdb msg="write block" mint=1527876000000 maxt=1527883200000 ulid=01CEYJCSWX1KT79AX0TAYQFC3H
level=info ts=2018-06-01T21:00:24.411146383Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.4009823s
level=info ts=2018-06-01T21:01:38.119337601Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1m13.700595502s
level=info ts=2018-06-01T21:01:42.808912311Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527854400000 maxt=1527876000000 ulid=01CEYJFV4JPXEHPTJFGB4FPQ3H sources="[01CEXXV8YTY3CC90NC5W46JBX2 01CEY4VTWPK0TDZJKB5J2YCXXE 01CEYBJGXK379EPSWJ3G0FVPCQ]"
level=info ts=2018-06-01T21:01:50.461817883Z caller=compact.go:387 component=tsdb msg="write block" mint=1527876000000 maxt=1527883200000 ulid=01CEYJG31THKSYVF4ZY5316FP8
level=info ts=2018-06-01T23:00:40.82446354Z caller=compact.go:387 component=tsdb msg="write block" mint=1527883200000 maxt=1527890400000 ulid=01CEYS8H4YTYMWFF569FXDN59K
level=info ts=2018-06-01T23:00:44.414928124Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.099206532s
level=info ts=2018-06-01T23:01:17.094926861Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=32.669268726s
level=info ts=2018-06-01T23:01:31.968470199Z caller=compact.go:387 component=tsdb msg="write block" mint=1527883200000 maxt=1527890400000 ulid=01CEYSB2NH8PCAJ946Z8DHXTE9
level=info ts=2018-06-02T01:00:14.544393044Z caller=compact.go:387 component=tsdb msg="write block" mint=1527890400000 maxt=1527897600000 ulid=01CEZ048D3QSM7PBBPW0G04VV1
level=info ts=2018-06-02T01:00:18.121210495Z caller=head.go:348 component=tsdb msg="head GC completed" duration=960.452443ms
level=info ts=2018-06-02T01:00:35.64195699Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=17.497041753s
level=info ts=2018-06-02T01:00:45.805477844Z caller=compact.go:387 component=tsdb msg="write block" mint=1527890400000 maxt=1527897600000 ulid=01CEZ05JQJ2TGAZ3EP0280AD2X
level=info ts=2018-06-02T03:00:39.583058453Z caller=compact.go:387 component=tsdb msg="write block" mint=1527897600000 maxt=1527904800000 ulid=01CEZ6ZZN0EB8QCCW058A5G3QA
level=info ts=2018-06-02T03:00:43.938032352Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.195611847s
level=info ts=2018-06-02T03:02:08.347812286Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1m24.383507816s
level=info ts=2018-06-02T03:02:18.088984565Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527876000000 maxt=1527897600000 ulid=01CEZ73Z1RSYP3P79N9JD5PH6D sources="[01CEYJG31THKSYVF4ZY5316FP8 01CEYSB2NH8PCAJ946Z8DHXTE9 01CEZ05JQJ2TGAZ3EP0280AD2X]"
level=info ts=2018-06-02T03:02:32.792713101Z caller=compact.go:387 component=tsdb msg="write block" mint=1527897600000 maxt=1527904800000 ulid=01CEZ74DTMT4A22X4SBMBY107R
level=info ts=2018-06-02T05:01:05.242170734Z caller=compact.go:387 component=tsdb msg="write block" mint=1527904800000 maxt=1527912000000 ulid=01CEZDVPWVEM44VS403303D052
level=info ts=2018-06-02T05:01:09.63225087Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.456391263s
level=info ts=2018-06-02T05:02:33.999988119Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1m24.352041169s
level=info ts=2018-06-02T05:02:50.105876406Z caller=compact.go:387 component=tsdb msg="write block" mint=1527904800000 maxt=1527912000000 ulid=01CEZE0QR80EV9C1W2GR7Y31GE
level=info ts=2018-06-02T07:01:24.153663895Z caller=compact.go:387 component=tsdb msg="write block" mint=1527912000000 maxt=1527919200000 ulid=01CEZMQE5CN357GHCYD2KFP092
level=info ts=2018-06-02T07:01:28.807012745Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.94225964s
level=info ts=2018-06-02T07:01:45.288023648Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=16.466188698s
level=info ts=2018-06-02T07:02:08.242356225Z caller=compact.go:387 component=tsdb msg="write block" mint=1527912000000 maxt=1527919200000 ulid=01CEZMV82E0JMH4MEFMJVTS59F
level=info ts=2018-06-02T09:00:27.085849425Z caller=compact.go:387 component=tsdb msg="write block" mint=1527919200000 maxt=1527926400000 ulid=01CEZVK5CYT58YKP84AJDD1JWG
level=info ts=2018-06-02T09:00:32.848628834Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.197281573s
level=info ts=2018-06-02T09:00:48.664147126Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=15.809207376s
level=info ts=2018-06-02T09:00:54.281305502Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527897600000 maxt=1527919200000 ulid=01CEZVMQDPXAPBPNZTCFFRV2X1 sources="[01CEZ74DTMT4A22X4SBMBY107R 01CEZE0QR80EV9C1W2GR7Y31GE 01CEZMV82E0JMH4MEFMJVTS59F]"
level=info ts=2018-06-02T09:01:05.148360434Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527854400000 maxt=1527919200000 ulid=01CEZVN0GTK0W7V97DA6B7G8GP sources="[01CEYJFV4JPXEHPTJFGB4FPQ3H 01CEZ73Z1RSYP3P79N9JD5PH6D 01CEZVMQDPXAPBPNZTCFFRV2X1]"
level=info ts=2018-06-02T09:01:16.077882787Z caller=compact.go:387 component=tsdb msg="write block" mint=1527919200000 maxt=1527926400000 ulid=01CEZVND3WS513X1Q8G6D51ECF
level=info ts=2018-06-02T11:00:26.09411923Z caller=compact.go:387 component=tsdb msg="write block" mint=1527926400000 maxt=1527933600000 ulid=01CF02EWMWDB1X5TVJZXJVS7VZ
level=info ts=2018-06-02T11:00:30.006902691Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.0889891s
level=info ts=2018-06-02T11:00:40.686730075Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=10.672494078s
level=info ts=2018-06-02T11:00:48.6125432Z caller=compact.go:387 component=tsdb msg="write block" mint=1527926400000 maxt=1527933600000 ulid=01CF02G9M2BYX1SPFTW7BHPKMZ
level=info ts=2018-06-02T13:00:06.544333226Z caller=compact.go:387 component=tsdb msg="write block" mint=1527933600000 maxt=1527940800000 ulid=01CF09AKWZET1SC35SEW72W9QC
level=info ts=2018-06-02T13:00:12.026437949Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.233467239s
level=info ts=2018-06-02T13:00:18.38674537Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=6.355247453s
level=info ts=2018-06-02T15:00:05.982117176Z caller=compact.go:387 component=tsdb msg="write block" mint=1527940800000 maxt=1527948000000 ulid=01CF0G6B4QK6HTAGKJWJVB6XS6
level=info ts=2018-06-02T15:00:09.503649051Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.1567175s
level=info ts=2018-06-02T15:00:10.991880164Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.488035568s
level=info ts=2018-06-02T15:00:15.951765179Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527919200000 maxt=1527940800000 ulid=01CF0G6QWJ25TR9DR19EYS8J99 sources="[01CEZVND3WS513X1Q8G6D51ECF 01CF02G9M2BYX1SPFTW7BHPKMZ 01CF09AKWZET1SC35SEW72W9QC]"
level=info ts=2018-06-02T17:00:05.523640399Z caller=compact.go:387 component=tsdb msg="write block" mint=1527948000000 maxt=1527955200000 ulid=01CF0Q22CX19H6WK2KVECVJ3SA
level=info ts=2018-06-02T17:00:08.710899031Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.110988475s
level=info ts=2018-06-02T17:00:12.084907131Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=3.368606419s
level=info ts=2018-06-02T19:01:00.159190779Z caller=compact.go:387 component=tsdb msg="write block" mint=1527955200000 maxt=1527962400000 ulid=01CF0XXSMX8KG2RPJN5YAPWCHQ
level=info ts=2018-06-02T19:01:04.618878464Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.039988244s
level=info ts=2018-06-02T19:01:25.283388344Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=20.652416517s
level=info ts=2018-06-02T19:01:33.867674516Z caller=compact.go:387 component=tsdb msg="write block" mint=1527955200000 maxt=1527962400000 ulid=01CF0Y0JAZYRVCVT67XEQ9CPH8
level=info ts=2018-06-02T21:00:30.345117648Z caller=compact.go:387 component=tsdb msg="write block" mint=1527962400000 maxt=1527969600000 ulid=01CF14SGX0A9Y630GK8MJXPXTH
level=info ts=2018-06-02T21:00:34.463510236Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.055708796s
level=info ts=2018-06-02T21:00:45.507538177Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=11.04007724s
level=info ts=2018-06-02T21:00:50.384994519Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527940800000 maxt=1527962400000 ulid=01CF14TZ604KKYTK03JJXFCD6D sources="[01CF0G6B4QK6HTAGKJWJVB6XS6 01CF0Q22CX19H6WK2KVECVJ3SA 01CF0Y0JAZYRVCVT67XEQ9CPH8]"
level=info ts=2018-06-02T21:01:00.447570195Z caller=compact.go:387 component=tsdb msg="write block" mint=1527962400000 maxt=1527969600000 ulid=01CF14V9C9DQQDERNHMFEY5CYG
level=info ts=2018-06-02T23:00:19.120634336Z caller=compact.go:387 component=tsdb msg="write block" mint=1527969600000 maxt=1527976800000 ulid=01CF1BN851NEBRMWXM6B8XV8DZ
level=info ts=2018-06-02T23:00:24.172025491Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.223674132s
level=info ts=2018-06-02T23:00:33.970498794Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=9.78444809s
level=info ts=2018-06-02T23:00:42.731208437Z caller=compact.go:387 component=tsdb msg="write block" mint=1527969600000 maxt=1527976800000 ulid=01CF1BPFCJZXMKV2HM9TYTSE1S
level=info ts=2018-06-03T01:00:33.896714267Z caller=compact.go:387 component=tsdb msg="write block" mint=1527976800000 maxt=1527984000000 ulid=01CF1JGZD7T6VJAP9AEQPEH98J
level=info ts=2018-06-03T01:00:39.684025245Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.177659231s
level=info ts=2018-06-03T01:00:49.674247168Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=9.977980598s
level=info ts=2018-06-03T01:00:58.625333268Z caller=compact.go:387 component=tsdb msg="write block" mint=1527976800000 maxt=1527984000000 ulid=01CF1JJNXZDR3R0HH0V9S4XQTJ
level=info ts=2018-06-03T03:00:06.997622062Z caller=compact.go:387 component=tsdb msg="write block" mint=1527984000000 maxt=1527991200000 ulid=01CF1SCPMRGTD13GRSFPMZAPZ6
level=info ts=2018-06-03T03:00:11.941771029Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.486452401s
level=info ts=2018-06-03T03:00:15.635448389Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=3.693553803s
level=info ts=2018-06-03T03:00:24.372111078Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527962400000 maxt=1527984000000 ulid=01CF1SDAV3YP9Y431QYDP2HW8G sources="[01CF14V9C9DQQDERNHMFEY5CYG 01CF1BPFCJZXMKV2HM9TYTSE1S 01CF1JJNXZDR3R0HH0V9S4XQTJ]"
level=info ts=2018-06-03T03:00:30.702363248Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527919200000 maxt=1527984000000 ulid=01CF1SDFP3R0XACYCQEYZNW6VM sources="[01CF0G6QWJ25TR9DR19EYS8J99 01CF14TZ604KKYTK03JJXFCD6D 01CF1SDAV3YP9Y431QYDP2HW8G]"
level=info ts=2018-06-03T05:00:01.203029347Z caller=compact.go:387 component=tsdb msg="write block" mint=1527991200000 maxt=1527998400000 ulid=01CF208DWRQ8NY8SWQM5R92ARH
level=info ts=2018-06-03T05:00:01.417878874Z caller=head.go:348 component=tsdb msg="head GC completed" duration=32.56797ms
level=info ts=2018-06-03T05:00:02.543590543Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.12557426s
level=info ts=2018-06-03T05:00:10.271193299Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527811200000 maxt=1527984000000 ulid=01CF208GKM2VGAFK0GVJ6GVEHX sources="[01CEY4TYHB1NX270JPQ3M6R74C 01CEZVN0GTK0W7V97DA6B7G8GP 01CF1SDFP3R0XACYCQEYZNW6VM]"
level=info ts=2018-06-03T07:00:01.217207416Z caller=compact.go:387 component=tsdb msg="write block" mint=1527998400000 maxt=1528005600000 ulid=01CF2745589WZ6TQQVRV20KNF5
level=info ts=2018-06-03T07:00:01.563154728Z caller=head.go:348 component=tsdb msg="head GC completed" duration=34.207224ms
level=info ts=2018-06-03T07:00:01.632363785Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=69.096068ms
level=info ts=2018-06-03T09:00:01.079308813Z caller=compact.go:387 component=tsdb msg="write block" mint=1528005600000 maxt=1528012800000 ulid=01CF2DZWCJA7AHKJKRSYZX6KK1
level=info ts=2018-06-03T09:00:01.276689835Z caller=head.go:348 component=tsdb msg="head GC completed" duration=25.644638ms
level=info ts=2018-06-03T09:00:01.750647675Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=473.818802ms
level=info ts=2018-06-03T09:00:03.516145331Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527984000000 maxt=1528005600000 ulid=01CF2DZXRN43RTEA8EXS4DV37V sources="[01CF1SCPMRGTD13GRSFPMZAPZ6 01CF208DWRQ8NY8SWQM5R92ARH 01CF2745589WZ6TQQVRV20KNF5]"
level=info ts=2018-06-03T11:00:01.097859357Z caller=compact.go:387 component=tsdb msg="write block" mint=1528012800000 maxt=1528020000000 ulid=01CF2MVKMJMWH31MJA440SJKWQ
level=info ts=2018-06-03T11:00:01.411708711Z caller=head.go:348 component=tsdb msg="head GC completed" duration=40.252634ms
level=info ts=2018-06-03T11:00:01.514320182Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=102.486293ms
level=info ts=2018-06-03T13:00:01.179562002Z caller=compact.go:387 component=tsdb msg="write block" mint=1528020000000 maxt=1528027200000 ulid=01CF2VQAWSFSYVX59F11AMDSQM
level=info ts=2018-06-03T13:00:01.474042099Z caller=head.go:348 component=tsdb msg="head GC completed" duration=28.220749ms
level=info ts=2018-06-03T13:00:01.978252512Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=504.084228ms
level=info ts=2018-06-03T15:00:01.197500915Z caller=compact.go:387 component=tsdb msg="write block" mint=1528027200000 maxt=1528034400000 ulid=01CF32K251GZBS1P4NTZZN1N00
level=info ts=2018-06-03T15:00:01.44010553Z caller=head.go:348 component=tsdb msg="head GC completed" duration=34.935555ms
level=info ts=2018-06-03T15:00:01.509549113Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=69.328796ms
level=info ts=2018-06-03T15:00:02.39437415Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1528005600000 maxt=1528027200000 ulid=01CF32K34D2DPW4RPGKB6ZMHCM sources="[01CF2DZWCJA7AHKJKRSYZX6KK1 01CF2MVKMJMWH31MJA440SJKWQ 01CF2VQAWSFSYVX59F11AMDSQM]"
level=info ts=2018-06-03T17:00:01.130894275Z caller=compact.go:387 component=tsdb msg="write block" mint=1528034400000 maxt=1528041600000 ulid=01CF39ESCMP7FB7NBGWY7ZEE17
level=info ts=2018-06-03T17:00:01.327769687Z caller=head.go:348 component=tsdb msg="head GC completed" duration=29.536917ms
level=info ts=2018-06-03T17:00:01.823412507Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=495.535304ms
level=info ts=2018-06-03T19:00:01.259109795Z caller=compact.go:387 component=tsdb msg="write block" mint=1528041600000 maxt=1528048800000 ulid=01CF3GAGMTZN1AXNG6KHA93XHK
level=info ts=2018-06-03T19:00:01.440289594Z caller=head.go:348 component=tsdb msg="head GC completed" duration=30.942554ms
level=info ts=2018-06-03T19:00:01.500771956Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=60.371634ms
level=info ts=2018-06-03T21:00:01.166353313Z caller=compact.go:387 component=tsdb msg="write block" mint=1528048800000 maxt=1528056000000 ulid=01CF3Q67WHPGGF6K2E9YHZXR6T
level=info ts=2018-06-03T21:00:01.591498135Z caller=head.go:348 component=tsdb msg="head GC completed" duration=30.778822ms
level=info ts=2018-06-03T21:00:01.654541913Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=62.9295ms
level=info ts=2018-06-03T21:00:02.421410115Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1528027200000 maxt=1528048800000 ulid=01CF3Q68YBMC7A1BSDJ2S8NEZH sources="[01CF32K251GZBS1P4NTZZN1N00 01CF39ESCMP7FB7NBGWY7ZEE17 01CF3GAGMTZN1AXNG6KHA93XHK]"
level=info ts=2018-06-03T21:00:04.71548136Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1527984000000 maxt=1528048800000 ulid=01CF3Q6A1FH460ASXTV10RRM4Z sources="[01CF2DZXRN43RTEA8EXS4DV37V 01CF32K34D2DPW4RPGKB6ZMHCM 01CF3Q68YBMC7A1BSDJ2S8NEZH]"
level=info ts=2018-06-03T23:00:01.193922843Z caller=compact.go:387 component=tsdb msg="write block" mint=1528056000000 maxt=1528063200000 ulid=01CF3Y1Z4VDKM3TK3FT3442VWS
level=info ts=2018-06-03T23:00:01.523929938Z caller=head.go:348 component=tsdb msg="head GC completed" duration=32.112061ms
level=info ts=2018-06-03T23:00:02.016591598Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=492.54338ms
level=info ts=2018-06-04T01:00:01.166596505Z caller=compact.go:387 component=tsdb msg="write block" mint=1528063200000 maxt=1528070400000 ulid=01CF44XPCTY2M7CVXC5VXBXTTE
level=info ts=2018-06-04T01:00:01.406025342Z caller=head.go:348 component=tsdb msg="head GC completed" duration=39.044544ms
level=info ts=2018-06-04T01:00:01.466115373Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=59.98812ms
level=warn ts=2018-06-04T01:01:11.46572337Z caller=main.go:373 msg="Received SIGTERM, exiting gracefully..."
level=info ts=2018-06-04T01:01:11.465835951Z caller=main.go:398 msg="Stopping scrape discovery manager..."
level=info ts=2018-06-04T01:01:11.465886753Z caller=main.go:412 msg="Stopping notify discovery manager..."
level=info ts=2018-06-04T01:01:11.46590945Z caller=main.go:434 msg="Stopping scrape manager..."
level=info ts=2018-06-04T01:01:11.465899805Z caller=main.go:394 msg="Scrape discovery manager stopped"
level=info ts=2018-06-04T01:01:11.465966061Z caller=main.go:408 msg="Notify discovery manager stopped"
level=info ts=2018-06-04T01:01:11.483196355Z caller=main.go:428 msg="Scrape manager stopped"
level=info ts=2018-06-04T01:01:11.504968038Z caller=manager.go:464 component="rule manager" msg="Stopping rule manager..."
level=info ts=2018-06-04T01:01:11.51802771Z caller=manager.go:470 component="rule manager" msg="Rule manager stopped"
level=info ts=2018-06-04T01:01:11.51813026Z caller=notifier.go:512 component=notifier msg="Stopping notification manager..."
level=info ts=2018-06-04T01:01:11.528202845Z caller=main.go:584 msg="Notifier manager stopped"
level=info ts=2018-06-04T01:01:11.531092233Z caller=main.go:595 msg="See you next time!"
level=info ts=2018-06-04T01:01:14.945378738Z caller=main.go:218 msg="Starting Prometheus" version="(version=2.2.1, branch=master, revision=7e376dfc8901baaa5efd4d7c2ab92ba46d041d7e)"
level=info ts=2018-06-04T01:01:14.945504256Z caller=main.go:219 build_context="(go=go1.10.2, user=root@94b12c4de194, date=20180529-13:25:24)"
level=info ts=2018-06-04T01:01:14.945551Z caller=main.go:220 host_details="(Linux 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 localhost.localdomain (none))"
level=info ts=2018-06-04T01:01:14.945589674Z caller=main.go:221 fd_limits="(soft=65536, hard=65536)"
level=info ts=2018-06-04T01:01:14.94708283Z caller=web.go:426 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-04T01:01:14.947019744Z caller=main.go:510 msg="Starting TSDB ..."
level=info ts=2018-06-04T01:01:14.947604877Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1527811200000 maxt=1527984000000 ulid=01CF208GKM2VGAFK0GVJ6GVEHX
level=info ts=2018-06-04T01:01:14.947741905Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528048800000 maxt=1528056000000 ulid=01CF3Q67WHPGGF6K2E9YHZXR6T
level=info ts=2018-06-04T01:01:14.947848651Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1527984000000 maxt=1528048800000 ulid=01CF3Q6A1FH460ASXTV10RRM4Z
level=info ts=2018-06-04T01:01:14.947945909Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528056000000 maxt=1528063200000 ulid=01CF3Y1Z4VDKM3TK3FT3442VWS
level=info ts=2018-06-04T01:01:14.948032672Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528063200000 maxt=1528070400000 ulid=01CF44XPCTY2M7CVXC5VXBXTTE
level=info ts=2018-06-04T01:01:17.931565688Z caller=main.go:520 msg="TSDB started"
level=info ts=2018-06-04T01:01:17.931748753Z caller=main.go:599 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2018-06-04T01:01:17.973460066Z caller=main.go:496 msg="Server is ready to receive web requests."
level=info ts=2018-06-04T03:00:03.619562161Z caller=compact.go:387 component=tsdb msg="write block" mint=1528070400000 maxt=1528077600000 ulid=01CF4BSFP0M50PBNJ4PA3F7V0M
level=info ts=2018-06-04T03:00:04.119389239Z caller=head.go:348 component=tsdb msg="head GC completed" duration=103.500196ms
level=info ts=2018-06-04T03:00:04.602316614Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=482.817324ms
level=info ts=2018-06-04T03:00:05.243202168Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1528048800000 maxt=1528070400000 ulid=01CF4BSHG035WXC1NFBA7ZS188 sources="[01CF3Q67WHPGGF6K2E9YHZXR6T 01CF3Y1Z4VDKM3TK3FT3442VWS 01CF44XPCTY2M7CVXC5VXBXTTE]"
level=info ts=2018-06-04T05:00:04.831321255Z caller=compact.go:387 component=tsdb msg="write block" mint=1528077600000 maxt=1528084800000 ulid=01CF4JN6Y14JNE2TX9PHE9Q2FQ
level=info ts=2018-06-04T05:00:05.543424177Z caller=head.go:348 component=tsdb msg="head GC completed" duration=214.344348ms
level=info ts=2018-06-04T05:00:06.296881048Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=753.336835ms
level=info ts=2018-06-04T07:00:05.114113314Z caller=compact.go:387 component=tsdb msg="write block" mint=1528084800000 maxt=1528092000000 ulid=01CF4SGY61V43XBFY6TACHA48B
level=info ts=2018-06-04T07:00:06.055064979Z caller=head.go:348 component=tsdb msg="head GC completed" duration=299.470879ms
level=info ts=2018-06-04T07:00:07.328333526Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=1.273159812s
level=info ts=2018-06-04T09:00:05.548908086Z caller=compact.go:387 component=tsdb msg="write block" mint=1528092000000 maxt=1528099200000 ulid=01CF50CNNVCWGRDVV6HRXBS207
level=info ts=2018-06-04T09:00:06.803780768Z caller=head.go:348 component=tsdb msg="head GC completed" duration=420.975923ms
level=info ts=2018-06-04T09:00:15.477875951Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=8.673984164s
level=info ts=2018-06-04T09:00:17.991485923Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1528070400000 maxt=1528092000000 ulid=01CF50D32CA25V7HGVCAC0FKV4 sources="[01CF4BSFP0M50PBNJ4PA3F7V0M 01CF4JN6Y14JNE2TX9PHE9Q2FQ 01CF4SGY61V43XBFY6TACHA48B]"
level=info ts=2018-06-04T11:00:07.157552276Z caller=compact.go:387 component=tsdb msg="write block" mint=1528099200000 maxt=1528106400000 ulid=01CF578D9FCGQ5B0BECVEGEZHW
level=info ts=2018-06-04T11:00:08.810304822Z caller=head.go:348 component=tsdb msg="head GC completed" duration=598.50415ms
level=info ts=2018-06-04T11:00:20.390205465Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=11.568423057s
level=info ts=2018-06-04T13:00:09.147888237Z caller=compact.go:387 component=tsdb msg="write block" mint=1528106400000 maxt=1528113600000 ulid=01CF5E445A9KDMZ1VSAEZ0ASYT
level=info ts=2018-06-04T13:00:11.647163377Z caller=head.go:348 component=tsdb msg="head GC completed" duration=583.280582ms
level=info ts=2018-06-04T13:00:23.591074767Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=11.94378862s
level=info ts=2018-06-04T13:00:26.495475035Z caller=compact.go:387 component=tsdb msg="write block" mint=1528106400000 maxt=1528113600000 ulid=01CF5E4T3DFP06GV6Q1WR43RHB
level=info ts=2018-06-04T15:00:10.397827397Z caller=compact.go:387 component=tsdb msg="write block" mint=1528113600000 maxt=1528120800000 ulid=01CF5MZVCCBK016C46N5BHC62R
level=info ts=2018-06-04T15:00:12.743881467Z caller=head.go:348 component=tsdb msg="head GC completed" duration=755.680171ms
level=info ts=2018-06-04T15:00:26.459131611Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=13.715111806s
level=info ts=2018-06-04T15:00:29.318301437Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1528092000000 maxt=1528113600000 ulid=01CF5N0KFX3WY2D4N8TDKKC49V sources="[01CF50CNNVCWGRDVV6HRXBS207 01CF578D9FCGQ5B0BECVEGEZHW 01CF5E4T3DFP06GV6Q1WR43RHB]"
level=info ts=2018-06-04T15:00:35.652206856Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1528048800000 maxt=1528113600000 ulid=01CF5N0S4139AZNY78SDF94J01 sources="[01CF4BSHG035WXC1NFBA7ZS188 01CF50D32CA25V7HGVCAC0FKV4 01CF5N0KFX3WY2D4N8TDKKC49V]"
level=info ts=2018-06-04T15:00:39.932415451Z caller=compact.go:387 component=tsdb msg="write block" mint=1528113600000 maxt=1528120800000 ulid=01CF5N0YBGW36WST8AN0AX2KNV
level=info ts=2018-06-04T17:00:09.161002449Z caller=compact.go:387 component=tsdb msg="write block" mint=1528120800000 maxt=1528128000000 ulid=01CF5VVJFVJCK5FZM0FTBE8J6K
level=info ts=2018-06-04T17:00:11.902779013Z caller=head.go:348 component=tsdb msg="head GC completed" duration=670.585307ms
level=info ts=2018-06-04T17:00:26.124467179Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=14.221575463s
level=info ts=2018-06-04T17:00:29.063401169Z caller=compact.go:387 component=tsdb msg="write block" mint=1528120800000 maxt=1528128000000 ulid=01CF5VWB041Y6E17BACPTKQFTH
level=info ts=2018-06-04T19:00:07.016535319Z caller=compact.go:387 component=tsdb msg="write block" mint=1528128000000 maxt=1528135200000 ulid=01CF62QAQMDJDV1D91CP6WMK91
level=info ts=2018-06-04T19:00:09.207131566Z caller=head.go:348 component=tsdb msg="head GC completed" duration=640.697052ms
level=info ts=2018-06-04T19:00:23.233915391Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=14.026674506s
level=info ts=2018-06-04T21:00:08.726846757Z caller=compact.go:387 component=tsdb msg="write block" mint=1528135200000 maxt=1528142400000 ulid=01CF69K11PRVH5R3W12WYCG0XK
level=info ts=2018-06-04T21:00:11.253523131Z caller=head.go:348 component=tsdb msg="head GC completed" duration=617.814557ms
level=info ts=2018-06-04T21:00:24.745621905Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=13.491968416s
level=info ts=2018-06-04T21:00:27.615170559Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1528113600000 maxt=1528135200000 ulid=01CF69KQJMTEMGRRJNVF7YXW9W sources="[01CF5N0YBGW36WST8AN0AX2KNV 01CF5VWB041Y6E17BACPTKQFTH 01CF62QAQMDJDV1D91CP6WMK91]"
level=info ts=2018-06-04T21:00:30.92369474Z caller=compact.go:387 component=tsdb msg="write block" mint=1528135200000 maxt=1528142400000 ulid=01CF69KVAEKF5R48G126K1RGZP
level=info ts=2018-06-04T23:00:10.264465237Z caller=compact.go:387 component=tsdb msg="write block" mint=1528142400000 maxt=1528149600000 ulid=01CF6GER97KKZA2Y02D4963HFN
level=info ts=2018-06-04T23:00:12.460396289Z caller=head.go:348 component=tsdb msg="head GC completed" duration=686.979307ms
level=info ts=2018-06-04T23:00:26.316720176Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=13.856196031s
level=info ts=2018-06-04T23:00:29.599140468Z caller=compact.go:387 component=tsdb msg="write block" mint=1528142400000 maxt=1528149600000 ulid=01CF6GFHAHWFJZTS75095PC18H
level=info ts=2018-06-05T01:00:10.742228778Z caller=compact.go:387 component=tsdb msg="write block" mint=1528149600000 maxt=1528156800000 ulid=01CF6QAFJV5K5MNVBBN7B9NPV2
level=info ts=2018-06-05T01:00:13.126518999Z caller=head.go:348 component=tsdb msg="head GC completed" duration=728.091985ms
level=info ts=2018-06-05T01:00:31.189548475Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=18.062918998s
level=info ts=2018-06-05T01:00:34.279380277Z caller=compact.go:387 component=tsdb msg="write block" mint=1528149600000 maxt=1528156800000 ulid=01CF6QBD4F1T58VPR3H0Z8R7M8
level=info ts=2018-06-05T03:00:11.485109459Z caller=compact.go:387 component=tsdb msg="write block" mint=1528156800000 maxt=1528164000000 ulid=01CF6Y6706MGYWY8RPN5JPK5SV
level=info ts=2018-06-05T03:00:14.404029824Z caller=head.go:348 component=tsdb msg="head GC completed" duration=701.319347ms
level=info ts=2018-06-05T03:00:30.735398873Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=16.331273231s
level=info ts=2018-06-05T03:00:33.803926165Z caller=compact.go:347 component=tsdb msg="compact blocks" count=3 mint=1528135200000 maxt=1528156800000 ulid=01CF6Y73B8GRCPEYAZP6XMNDEQ sources="[01CF69KVAEKF5R48G126K1RGZP 01CF6GFHAHWFJZTS75095PC18H 01CF6QBD4F1T58VPR3H0Z8R7M8]"
level=info ts=2018-06-05T03:00:38.245854594Z caller=compact.go:387 component=tsdb msg="write block" mint=1528156800000 maxt=1528164000000 ulid=01CF6Y78638T7PW5P7AMS4FR93
level=info ts=2018-06-05T05:00:12.57427237Z caller=compact.go:387 component=tsdb msg="write block" mint=1528164000000 maxt=1528171200000 ulid=01CF751YENJ0BTX69SB6YQZF6A
level=info ts=2018-06-05T05:00:15.892916411Z caller=head.go:348 component=tsdb msg="head GC completed" duration=974.10225ms
level=info ts=2018-06-05T05:00:38.497786774Z caller=head.go:357 component=tsdb msg="WAL truncation completed" duration=22.604755364s
level=info ts=2018-06-05T05:00:42.785264414Z caller=compact.go:387 component=tsdb msg="write block" mint=1528164000000 maxt=1528171200000 ulid=01CF7533PYJZ89AJSYZTV2CBGQ
level=info ts=2018-06-05T07:00:26.87704425Z caller=compact.go:387 component=tsdb msg="write block" mint=1528171200000 maxt=1528178400000 ulid=01CF7BXPF2N9NPXXGRF3PVKJ9E
level=info ts=2018-06-05T07:00:31.143339137Z caller=head.go:348 component=tsdb msg="head GC completed" duration=1.375337015s
level=error ts=2018-06-05T07:02:18.492813047Z caller=notifier.go:473 component=notifier alertmanager=http://localhost:9093/api/v1/alerts count=1 msg="Error sending alert" err="context deadline exceeded"
level=error ts=2018-06-05T07:04:03.16503332Z caller=notifier.go:473 component=notifier alertmanager=http://localhost:9093/api/v1/alerts count=2 msg="Error sending alert" err="context deadline exceeded"
level=error ts=2018-06-05T07:24:03.489890331Z caller=notifier.go:473 component=notifier alertmanager=http://localhost:9093/api/v1/alerts count=1 msg="Error sending alert" err="context deadline exceeded"
level=error ts=2018-06-05T07:25:46.023301109Z caller=notifier.go:473 component=notifier alertmanager=http://localhost:9093/api/v1/alerts count=4 msg="Error sending alert" err="context deadline exceeded"
level=error ts=2018-06-05T07:28:16.787781793Z caller=notifier.go:473 component=notifier alertmanager=http://localhost:9093/api/v1/alerts count=2 msg="Error sending alert" err="context deadline exceeded"
level=info ts=2018-06-05T07:35:44.912398286Z caller=main.go:218 msg="Starting Prometheus" version="(version=2.2.1, branch=master, revision=7e376dfc8901baaa5efd4d7c2ab92ba46d041d7e)"
level=info ts=2018-06-05T07:35:44.912529239Z caller=main.go:219 build_context="(go=go1.10.2, user=root@94b12c4de194, date=20180529-13:25:24)"
level=info ts=2018-06-05T07:35:44.912573125Z caller=main.go:220 host_details="(Linux 3.10.0-693.11.1.el7.x86_64 #1 SMP Mon Dec 4 23:52:40 UTC 2017 x86_64 localhost.localdomain (none))"
level=info ts=2018-06-05T07:35:44.912635184Z caller=main.go:221 fd_limits="(soft=65536, hard=65536)"
level=info ts=2018-06-05T07:35:44.914583068Z caller=web.go:426 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2018-06-05T07:35:44.914281083Z caller=main.go:510 msg="Starting TSDB ..."
level=info ts=2018-06-05T07:35:44.930529608Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1527811200000 maxt=1527984000000 ulid=01CF208GKM2VGAFK0GVJ6GVEHX
level=info ts=2018-06-05T07:35:44.937922123Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1527984000000 maxt=1528048800000 ulid=01CF3Q6A1FH460ASXTV10RRM4Z
level=info ts=2018-06-05T07:35:44.943029228Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528048800000 maxt=1528113600000 ulid=01CF5N0S4139AZNY78SDF94J01
level=info ts=2018-06-05T07:35:44.950466023Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528113600000 maxt=1528135200000 ulid=01CF69KQJMTEMGRRJNVF7YXW9W
level=info ts=2018-06-05T07:35:44.956027098Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528135200000 maxt=1528156800000 ulid=01CF6Y73B8GRCPEYAZP6XMNDEQ
level=info ts=2018-06-05T07:35:44.96268124Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528156800000 maxt=1528164000000 ulid=01CF6Y78638T7PW5P7AMS4FR93
level=info ts=2018-06-05T07:35:44.969820272Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528164000000 maxt=1528171200000 ulid=01CF7533PYJZ89AJSYZTV2CBGQ
level=info ts=2018-06-05T07:35:44.975148783Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1528171200000 maxt=1528178400000 ulid=01CF7BXPF2N9NPXXGRF3PVKJ9E
level=info ts=2018-06-05T07:38:51.484277436Z caller=main.go:520 msg="TSDB started"
level=info ts=2018-06-05T07:38:51.484431613Z caller=main.go:599 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2018-06-05T07:38:51.533275987Z caller=main.go:496 msg="Server is ready to receive web requests."
level=info ts=2018-06-05T07:40:05.432114317Z caller=compact.go:387 component=tsdb msg="write block" mint=1528171200000 maxt=1528178400000 ulid=01CF7E6Y6SCV88MBB6VK63WY1E
@codwu

This comment has been minimized.

Sign in to view
Copy link

codwu commented Jun 5, 2018

It seems that the error "context deadline exceeded" is cause by sending alert to AlertManager, not related to this issue.
@krasi-georgiev

This comment has been minimized.

Sign in to view
Copy link
Member

krasi-georgiev commented Jun 5, 2018

the only unusual log here is

level=info ts=2018-06-04T01:01:14.947604877Z caller=repair.go:39 component=tsdb msg="found healthy block" mint=1527811200000 maxt=1527984000000 ulid=01CF208GKM2VGAFK0GVJ6GVEHX

One other thing to be careful about is running more than one replica. If you mount the data folder from the host and run more than one replica at any given time this will cause 2 or more Prometheus instances to write to the same data folder which leads to many problems.

Are there no other logs related to the crash?
could it be OOM killed for using too much memory?
I am wondering if you have some expensive rules in

- /etc/prometheus/alert.rules

@fabxc fabxc closed this in prometheus/tsdb#330 Jul 11, 2018

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 22, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 22, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.