Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fatal Error : concurrent map read and map write #1786

Closed
prfalken opened this Issue Jul 4, 2016 · 23 comments

Comments

Assignees
No one assigned
Labels
kind/bug
Projects
None yet
Milestone
No milestone
5 participants
@prfalken @juliusv @brian-brazil @fabxc @beorn7
@prfalken
Copy link
Contributor

prfalken commented Jul 4, 2016

Runing prometheus for a while, 1.2M metrics.
The server randomly crashes with a huge trace. Occurs about every 10 hours or so.
Note : This is a federation server but the crash occured the same way when there was a single server scraping everything

process runs in an RKT container on Linux CoreOS (Linux 4.5.0-coreos-r1 x86_64)

  • Prometheus version:
    prometheus, version 0.19.2 (branch: master, revision: 23ca13c)
    build user: root@134dc6bbc274
    build date: 20160529-18:58:00
    go version: go1.6.2
  • Prometheus configuration file:
global:
  evaluation_interval: 20s
  scrape_interval:     20s
  scrape_timeout:      20s


scrape_configs:
  - job_name: 'prometheus'
    target_groups:
      - targets: ['127.0.0.1:9090']

  - job_name: federation
    metrics_path: /federate
    scrape_interval: 20s
    honor_labels: true
    nerve_sd_configs:
      - servers:
        - zk1
        - zk2
        - zk3
        paths:
          - /services/prometheus/prometheus
    params:
      'match[]':
         - '{env="prod"}'

    relabel_configs:
      - source_labels: [__meta_nerve_endpoint_name]
        target_label: name

      - source_labels: [__meta_nerve_endpoint_name]
        regex: ([a-z\-]+).*
        replacement: ${1}
        target_label: service

      - source_labels: [__meta_nerve_endpoint_name]
        regex: (r\d+-srv\d+)
        replacement: host
        target_label: service

      - source_labels: [job]
        regex: discovery_([a-z0-9\-]+)
        replacement: ${1}
        target_label: env

rule_files:
  - /etc/prometheus/rules/rules
  • Logs:
fatal error: concurrent map read and map write
goroutine 332096 [running]:
runtime.throw(0x1001ce0, 0x21)
/usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc98e9f7100 sp=0xc98e9f70e8
runtime.mapaccess1_faststr(0xe68260, 0xc82bcaf620, 0xc9097613c2, 0x3, 0xc8f7efe1a0)
/usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc98e9f7160 sp=0xc98e9f7100
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.SignatureForLabels(0xc82bcaf620, 0xc9a3a97180, 0x3, 0x4, 0xc86e5766af112801)
/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/signature.go:111 +0x1bb fp=0xc98e9f7220 sp=0xc98e9f7160
github.com/prometheus/prometheus/promql.(*evaluator).aggregation(0xc9415590c0, 0x28, 0xc9a3a97180, 0x3, 0x4, 0x0, 0xcab131c000, 0xbd00, 0xc000, 0x0, ...)
/go/src/github.com/prometheus/prometheus/promql/engine.go:1082 +0xc7c fp=0xc98e9f7450 sp=0xc98e9f7220
github.com/prometheus/prometheus/promql.(*evaluator).eval(0xc9415590c0, 0x7f8775541548, 0xc9a3a971c0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/promql/engine.go:613 +0x744 fp=0xc98e9f7640 sp=0xc98e9f7450
github.com/prometheus/prometheus/promql.(*evaluator).Eval(0xc9415590c0, 0x7f8775541548, 0xc9a3a971c0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/promql/engine.go:599 +0x9a fp=0xc98e9f7680 sp=0xc98e9f7640
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc8202c8c80, 0x7f876d67a320, 0xc909761530, 0xc9a3a97200, 0xc9097614a0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/promql/engine.go:404 +0x5e6 fp=0xc98e9f79f0 sp=0xc98e9f7680
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc8202c8c80, 0xc9a3a97200, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/promql/engine.go:357 +0x4ca fp=0xc98e9f7b08 sp=0xc98e9f79f0
github.com/prometheus/prometheus/promql.(*query).Exec(0xc9a3a97200, 0xc9097613b0)
/go/src/github.com/prometheus/prometheus/promql/engine.go:195 +0x2e fp=0xc98e9f7b68 sp=0xc98e9f7b08
github.com/prometheus/prometheus/rules.RecordingRule.eval(0xc900f476c0, 0x8, 0x7f8775541548, 0xca2a8c3980, 0x0, 0x155b4feb074, 0xc8202c8c80, 0x0, 0x0, 0x0, ...)
/go/src/github.com/prometheus/prometheus/rules/recording.go:55 +0x130 fp=0xc98e9f7d28 sp=0xc98e9f7b68
github.com/prometheus/prometheus/rules.(*RecordingRule).eval(0xc94fcd9170, 0x155b4feb074, 0xc8202c8c80, 0x0, 0x0, 0x0, 0x0, 0x0)
<autogenerated>:7 +0xd8 fp=0xc98e9f7d90 sp=0xc98e9f7d28
github.com/prometheus/prometheus/rules.(*Group).eval.func1(0xc9295fb220, 0xf1bf30, 0x9, 0x155b4feb074, 0xc99d667590, 0x7f8775541a18, 0xc94fcd9170)
/go/src/github.com/prometheus/prometheus/rules/manager.go:260 +0x1cc fp=0xc98e9f7f58 sp=0xc98e9f7d90
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc98e9f7f60 sp=0xc98e9f7f58
created by github.com/prometheus/prometheus/rules.(*Group).eval
/go/src/github.com/prometheus/prometheus/rules/manager.go:298 +0x170
goroutine 1 [select, 628 minutes]:
main.Main(0x0)
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:186 +0x1ed5
main.main()
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:41 +0x18
goroutine 22 [syscall, 3 minutes]:
os/signal.signal_recv(0x7f8775629078)
/usr/local/go/src/runtime/sigqueue.go:116 +0x132
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
/usr/local/go/src/os/signal/signal_unix.go:28 +0x37
goroutine 217 [select, 628 minutes, locked to thread]:
runtime.gopark(0x1101ee8, 0xc82048af28, 0xefe178, 0x6, 0x18, 0x2)
/usr/local/go/src/runtime/proc.go:262 +0x163
runtime.selectgoImpl(0xc82048af28, 0x0, 0x18)
/usr/local/go/src/runtime/select.go:392 +0xa67
runtime.selectgo(0xc82048af28)
/usr/local/go/src/runtime/select.go:215 +0x12
runtime.ensureSigM.func1()
/usr/local/go/src/runtime/signal1_unix.go:279 +0x358
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1998 +0x1
goroutine 329917 [semacquire]:
sync.runtime_Semacquire(0xc9295fb22c)
/usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc9295fb220)
/usr/local/go/src/sync/waitgroup.go:127 +0xb4
github.com/prometheus/prometheus/rules.(*Group).eval(0xc99d667590)
/go/src/github.com/prometheus/prometheus/rules/manager.go:300 +0x19d
github.com/prometheus/prometheus/rules.(*Group).run.func1()
/go/src/github.com/prometheus/prometheus/rules/manager.go:154 +0xbc
github.com/prometheus/prometheus/rules.(*Group).run(0xc99d667590)
/go/src/github.com/prometheus/prometheus/rules/manager.go:172 +0x25e
github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1.1(0xc82032a480, 0xc99d667590)
/go/src/github.com/prometheus/prometheus/rules/manager.go:468 +0x48
created by github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1
/go/src/github.com/prometheus/prometheus/rules/manager.go:469 +0xa0
goroutine 218 [select, 3 minutes]:
main.Main.func2(0xc82029be60, 0xc82029be00, 0xc82000a600, 0xc820086a40, 0x4, 0x4)
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:132 +0x135
created by main.Main
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:138 +0x1292
goroutine 219 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201408c0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270
goroutine 220 [select, 2 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc8200ce840)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x787
goroutine 221 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc8200ce840)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14c
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a9
goroutine 222 [select, 2 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc8200ce840)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7e0
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9ad
goroutine 223 [select, 6 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc8200ce840)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x254
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9cf
goroutine 224 [select, 641 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc8200ce840)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a3
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9f1
goroutine 225 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc820140a80)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270
goroutine 226 [select, 2 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc8200cef20)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x787
goroutine 227 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc8200cef20)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14c
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a9
goroutine 228 [select, 2 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc8200cef20)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7e0
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9ad
goroutine 229 [select, 11 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc8200cef20)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x254
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9cf
goroutine 230 [select, 641 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc8200cef20)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a3
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9f1
goroutine 231 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc820140700)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270
goroutine 232 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc8200cec60)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x787
goroutine 233 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc8200cec60)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14c
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a9
goroutine 234 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).pauseCompaction(0xc8200cec60, 0xc9c86f7a40)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:608 +0xf2
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc8200cec60)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:756 +0x528
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9ad
goroutine 235 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/golang/snappy.emitLiteral(0xcaf4bc40f8, 0x3a9d34f, 0x3a9d34f, 0xcb167340cf, 0x790da, 0x35aa2ab, 0x6a5f9)
/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/snappy/encode.go:16
github.com/prometheus/prometheus/vendor/github.com/golang/snappy.Encode(0xcaf480c000, 0x3e55447, 0x3e55447, 0xcb1637c000, 0xefc402, 0x396237a, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/golang/snappy/encode.go:130 +0x489
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Writer).writeBlock(0xc965186600, 0xc965186658, 0x2, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table/writer.go:170 +0x1d2
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Writer).finishBlock(0xc965186600, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table/writer.go:221 +0x75
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.(*Writer).Append(0xc965186600, 0xcb5f5e4dc5, 0x10, 0x1536523b, 0xcb5f5e4dd5, 0xefc3e4, 0x1536522b, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table/writer.go:254 +0x471
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*tWriter).append(0xcb876ef320, 0xcb5f5e4dc5, 0x10, 0x1536523b, 0xcb5f5e4dd5, 0xefc3e4, 0x1536522b, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.go:487 +0x2eb
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*tOps).createFrom(0xc8217e0510, 0x7f8775647620, 0xc9881b3880, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/table.go:325 +0x1c5
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*session).flushMemdb(0xc82051a160, 0xcb1a5294a0, 0xc8cdaba000, 0xffffffffffffffff, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/session_compaction.go:27 +0x19b
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).memCompaction.func1(0xca1018aac0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:274 +0x8e
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*compactionTransactFunc).run(0xca1018aab0, 0xca1018aac0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:220 +0x36
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0xc8200cec60, 0xf18a00, 0xb, 0x7f87755eacd0, 0xca1018aab0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:164 +0x27b
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransactFunc(0xc8200cec60, 0xf18a00, 0xb, 0xcb1b7981e0, 0xc8f8584560)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:231 +0xbe
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).memCompaction(0xc8200cec60)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:286 +0x53c
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc8200cec60)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:719 +0x180
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9cf
goroutine 236 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc8200cec60)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a3
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9f1
goroutine 82 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc82198a000)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270
goroutine 83 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc82199e000)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x787
goroutine 84 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc82199e000)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14c
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a9
goroutine 85 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc82199e000)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7e0
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9ad
goroutine 86 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc82199e000)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x254
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9cf
goroutine 87 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc82199e000)
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a3
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
/go/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9f1
goroutine 88 [runnable]:
github.com/prometheus/prometheus/storage/local/codable.(*FingerprintSet).UnmarshalBinary(0xc915f4fd60, 0xc9bf32a000, 0x3f529bc, 0x3f54000, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/codable/codable.go:248 +0x34b
github.com/prometheus/prometheus/storage/local/index.(*LevelDB).Get(0xc82133c040, 0x7f876dbac478, 0xc925c691a0, 0x7f876dbac450, 0xc915f4fd60, 0x7f876dbac478, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/index/leveldb.go:98 +0x202
github.com/prometheus/prometheus/storage/local/index.(*LabelPairFingerprintIndex).LookupSet(0xc8217ea040, 0xc8840cea98, 0x7, 0xc8840ceaa0, 0x4, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/index/index.go:238 +0x15a
github.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc820224b00)
/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:1291 +0x5b7
github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc820224b00)
/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:283 +0x21
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:330 +0x14c
goroutine 255 [runnable]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc820269900, 0xacbd6, 0x3ff0000000000000, 0xacb01)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:929 +0x19b
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints.func1(0xc95bff6660, 0xc820269900)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1013 +0x41e
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1022 +0x5d
goroutine 323 [chan send, 628 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 329920 [IO wait]:
net.runtime_pollWait(0x7f876d6fca38, 0x72, 0xcba4998000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8d62b3870, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8d62b3870, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8d62b3810, 0xcba4998000, 0x4, 0x180000, 0x0, 0x7f8775629050, 0xc82000e150)
/usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc941425100, 0xcba4998000, 0x4, 0x180000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
io.ReadAtLeast(0x7f87753f3820, 0xc941425100, 0xcba4998000, 0x4, 0x180000, 0x4, 0x0, 0x0, 0x0)
/usr/local/go/src/io/io.go:297 +0xe6
io.ReadFull(0x7f87753f3820, 0xc941425100, 0xcba4998000, 0x4, 0x180000, 0x0, 0x0, 0x0)
/usr/local/go/src/io/io.go:315 +0x62
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).recvLoop(0xc911dc4620, 0x7f87753f3770, 0xc941425100, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:544 +0x20f
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop.func2(0xc8ad730a10, 0xc911dc4620, 0xc8d6df11a0, 0xc8ad730b20)
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:279 +0x46
created by github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:285 +0x667
goroutine 254 [runnable]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc820269900, 0x169d26, 0x3fe6d57555555556, 0x1)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:929 +0x19b
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints.func1(0xc95bff6600, 0xc820269900)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:969 +0x250
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:979 +0x5d
goroutine 124 [select, 628 minutes]:
github.com/prometheus/prometheus/retrieval.(*targetSet).runScraping(0xc923f8f090, 0x7f8775632620, 0xc9b49cb340)
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:238 +0x2bb
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0xc8202ea9a0, 0xc923f8f090)
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:118 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:120 +0x20f
goroutine 359 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleEvictList(0xc820269900)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:825 +0x53a
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:353 +0x4b8
goroutine 360 [select, 628 minutes]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleQuarantine(0xc820269900)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1435 +0x514
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:354 +0x4dd
goroutine 361 [select, 628 minutes]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).logThrottling(0xc820269900)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:698 +0x8e2
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:355 +0x502
goroutine 362 [select]:
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loop(0xc820269900)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:1042 +0x5d6
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:356 +0x527
goroutine 367 [select, 4 minutes]:
github.com/prometheus/prometheus/notifier.(*Notifier).Run(0xc82000a540)
/go/src/github.com/prometheus/prometheus/notifier/notifier.go:169 +0x59a
created by main.Main
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:166 +0x167c
goroutine 433 [semacquire, 628 minutes]:
sync.runtime_Semacquire(0xc8202eaa04)
/usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc8202ea9f8)
/usr/local/go/src/sync/waitgroup.go:127 +0xb4
github.com/prometheus/prometheus/retrieval.(*TargetManager).Run(0xc8202ea9a0)
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:83 +0x209
created by main.Main
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:172 +0x171f
goroutine 434 [IO wait]:
net.runtime_pollWait(0x7f876d6fd3f8, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc941c287d0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc941c287d0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc941c28770, 0x0, 0x7f876d6fd4f0, 0xca160ff2e0)
/usr/local/go/src/net/fd_unix.go:426 +0x27c
net.(*TCPListener).AcceptTCP(0xc97cce2c78, 0x45a730, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc97cce2c78, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2427 +0x41
net/http.(*Server).Serve(0xc99dacf000, 0x7f876d6fd4b8, 0xc97cce2c78, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2117 +0x129
net/http.(*Server).ListenAndServe(0xc99dacf000, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2098 +0x136
net/http.ListenAndServe(0xef4e40, 0x5, 0x7f876d6fc420, 0xc820279a80, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2195 +0x98
github.com/prometheus/prometheus/web.(*Handler).Run(0xc82000a600)
/go/src/github.com/prometheus/prometheus/web/web.go:242 +0x144
created by main.Main
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:179 +0x179a
goroutine 329854 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954b40, 0x4a817c800, 0x4a817c800, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:433 +0x6ea
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c
goroutine 450 [select, 43 minutes]:
github.com/prometheus/prometheus/retrieval.(*targetSet).runScraping(0xc8f4a4a140, 0x7f8775632620, 0xc99ee1e280)
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:238 +0x2bb
github.com/prometheus/prometheus/retrieval.(*TargetManager).reload.func1(0xc8202ea9a0, 0xc8f4a4a140)
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:118 +0x43
created by github.com/prometheus/prometheus/retrieval.(*TargetManager).reload
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:120 +0x20f
goroutine 455 [chan send, 628 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 456 [chan send, 628 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 457 [chan send, 628 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 420 [chan send, 628 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 421 [chan send, 628 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 329629 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954c80, 0x4a817c800, 0x4a817c800, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:433 +0x6ea
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c
goroutine 329630 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.LabelSet.Equal(0xc9690db9b0, 0xc944fddb90, 0xe80b01)
/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/labelset.go:49 +0x1a4
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model.Metric.Equal(0xc9690db9b0, 0xc944fddb90, 0xc9a3fd4380)
/go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/model/metric.go:34 +0x2b
github.com/prometheus/prometheus/storage/local.(*fpMapper).mapFP(0xc908b41180, 0x614909a940e372fd, 0xc9690db9b0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/mapper.go:100 +0x93
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Append(0xc820269900, 0xc9c7b864e0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:622 +0x1b3
github.com/prometheus/prometheus/storage.Fanout.Append(0xc8202b32d0, 0x1, 0x1, 0xc9c7b864e0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/storage.go:60 +0xc8
github.com/prometheus/prometheus/storage.(*Fanout).Append(0xc8202799c0, 0xc9c7b864e0, 0x0, 0x0)
<autogenerated>:3 +0xb7
github.com/prometheus/prometheus/retrieval.honorLabelsAppender.Append(0x7f8775632690, 0xc8202799c0, 0xc910b45290, 0xc9c7b864e0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/target.go:264 +0x1d0
github.com/prometheus/prometheus/retrieval.(*honorLabelsAppender).Append(0xc9b2e03fe0, 0xc9c7b864e0, 0x0, 0x0)
<autogenerated>:30 +0xb7
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc8f1954cd0, 0xc99a020000, 0x15775, 0x17800)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:453 +0xc7
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954cd0, 0x4a817c800, 0x4a817c800, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:422 +0x539
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c
goroutine 330040 [select]:
github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp.Do(0x7f876d67a320, 0xca83e33050, 0xc910b451a0, 0xc97b21d260, 0xcc, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:56 +0x4bc
github.com/prometheus/prometheus/retrieval.(*targetScraper).scrape(0xc8ae7d6000, 0x7f876d67a320, 0xca83e33050, 0xecf0c1344, 0x7f8732153743, 0x1500ba0, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:320 +0x185
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954d70, 0x4a817c800, 0x4a817c800, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:420 +0x4c6
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c
goroutine 332021 [IO wait]:
net.runtime_pollWait(0x7f876d6fc978, 0x77, 0x5c2e34)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8ec9b1e90, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc8ec9b1e90, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc8ec9b1e30, 0x0, 0x0, 0x7f87753f37d0, 0xca1fb36a60, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:127 +0x28e
net.(*netFD).dial(0xc8ec9b1e30, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc92e764d20, 0x0, 0xc900000000, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:137 +0x364
net.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xc92e764d00, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc92e764d20, ...)
/usr/local/go/src/net/sock_posix.go:89 +0x429
net.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc92e764d20, 0x0, 0x0, 0x0, 0x1, ...)
/usr/local/go/src/net/ipsock_posix.go:161 +0x153
net.dialTCP(0xefebb8, 0x3, 0x0, 0xc92e764d20, 0x0, 0xc900000000, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b
net.dialSingle(0xc95884e120, 0x7f876d6fc448, 0xc92e764d20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:371 +0x40c
net.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:343 +0x75
net.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xc92e764d20, 0xca1fb36a40, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc95884e120, 0xc935b9a670, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:345 +0x7d0
net.(*Dialer).Dial(0xc86bdc9868, 0xefebb8, 0x3, 0xc99a93f187, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:239 +0x512
net.Dial(0xefebb8, 0x3, 0xc99a93f187, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:193 +0x96
net/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc99a93f187, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:665 +0x1d3
net/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc99a93f180, 0x4, 0xc99a93f187, 0x12, 0xc910b451a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:762 +0x1e3d
net/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc99a93f180, 0x4, 0xc99a93f187, 0x12, 0xc9387b2ba0)
/usr/local/go/src/net/http/transport.go:706 +0x66
created by net/http.(*Transport).getConn
/usr/local/go/src/net/http/transport.go:708 +0x262
goroutine 329913 [select, 3 minutes]:
github.com/prometheus/prometheus/retrieval.(*targetSet).runProviders.func1(0xc963da72f0, 0xca189398c0, 0xc8f4a4a140, 0xc963da7310, 0xc963da72d0, 0x7, 0x7f87755413d8, 0xc9793b6720)
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:310 +0x465
created by github.com/prometheus/prometheus/retrieval.(*targetSet).runProviders
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:326 +0x2bc
goroutine 331235 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7f876a68bc50, 0x77, 0x5c2e34)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc9490b0760, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc9490b0760, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc9490b0700, 0x0, 0x0, 0x7f87753f37d0, 0xc967b74a60, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:127 +0x28e
net.(*netFD).dial(0xc9490b0700, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca49a09860, 0x0, 0xc900000000, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:137 +0x364
net.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xca49a09800, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca49a09860, ...)
/usr/local/go/src/net/sock_posix.go:89 +0x429
net.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca49a09860, 0x0, 0x0, 0x0, 0x1, ...)
/usr/local/go/src/net/ipsock_posix.go:161 +0x153
net.dialTCP(0xefebb8, 0x3, 0x0, 0xca49a09860, 0x0, 0xc900000000, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b
net.dialSingle(0xc9a2ede090, 0x7f876d6fc448, 0xca49a09860, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:371 +0x40c
net.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:343 +0x75
net.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xca49a09860, 0xc967b749e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc9a2ede090, 0xc93040cb00, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:345 +0x7d0
net.(*Dialer).Dial(0xc96dced868, 0xefebb8, 0x3, 0xc8d8152be7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:239 +0x512
net.Dial(0xefebb8, 0x3, 0xc8d8152be7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:193 +0x96
net/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc8d8152be7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:665 +0x1d3
net/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc8d8152be0, 0x4, 0xc8d8152be7, 0x12, 0xc910b451a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:762 +0x1e3d
net/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc8d8152be0, 0x4, 0xc8d8152be7, 0x12, 0xc956e44b40)
/usr/local/go/src/net/http/transport.go:706 +0x66
created by net/http.(*Transport).getConn
/usr/local/go/src/net/http/transport.go:708 +0x262
goroutine 331718 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7f876d6fc4f8, 0x77, 0x5c2e34)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc91813bfe0, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc91813bfe0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc91813bf80, 0x0, 0x0, 0x7f87753f37d0, 0xc950de53e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:127 +0x28e
net.(*netFD).dial(0xc91813bf80, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc8cef5d470, 0x0, 0xc800000000, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:137 +0x364
net.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xc8cef5d400, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc8cef5d470, ...)
/usr/local/go/src/net/sock_posix.go:89 +0x429
net.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc8cef5d470, 0x0, 0x0, 0x0, 0x1, ...)
/usr/local/go/src/net/ipsock_posix.go:161 +0x153
net.dialTCP(0xefebb8, 0x3, 0x0, 0xc8cef5d470, 0x0, 0xc800000000, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b
net.dialSingle(0xc918135830, 0x7f876d6fc448, 0xc8cef5d470, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:371 +0x40c
net.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:343 +0x75
net.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xc8cef5d470, 0xc950de53c0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc918135830, 0xc8c7a57540, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:345 +0x7d0
net.(*Dialer).Dial(0xc91fd3f868, 0xefebb8, 0x3, 0xcb1b55b5e7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:239 +0x512
net.Dial(0xefebb8, 0x3, 0xcb1b55b5e7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:193 +0x96
net/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xcb1b55b5e7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:665 +0x1d3
net/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xcb1b55b5e0, 0x4, 0xcb1b55b5e7, 0x12, 0xc910b451a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:762 +0x1e3d
net/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xcb1b55b5e0, 0x4, 0xcb1b55b5e7, 0x12, 0xc9602e5e60)
/usr/local/go/src/net/http/transport.go:706 +0x66
created by net/http.(*Transport).getConn
/usr/local/go/src/net/http/transport.go:708 +0x262
goroutine 332215 [IO wait]:
net.runtime_pollWait(0x7f876d6fc678, 0x77, 0x5c2e34)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc92f246370, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc92f246370, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc92f246310, 0x0, 0x0, 0x7f87753f37d0, 0xcacd9dc300, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:127 +0x28e
net.(*netFD).dial(0xc92f246310, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca83e33170, 0x0, 0xc800000000, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:137 +0x364
net.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xca83e33100, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca83e33170, ...)
/usr/local/go/src/net/sock_posix.go:89 +0x429
net.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xca83e33170, 0x0, 0x0, 0x0, 0x1, ...)
/usr/local/go/src/net/ipsock_posix.go:161 +0x153
net.dialTCP(0xefebb8, 0x3, 0x0, 0xca83e33170, 0x0, 0xc800000000, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b
net.dialSingle(0xc944bf2fc0, 0x7f876d6fc448, 0xca83e33170, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:371 +0x40c
net.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:343 +0x75
net.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xca83e33170, 0xcacd9dc2c0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc944bf2fc0, 0xc8f34e0720, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:345 +0x7d0
net.(*Dialer).Dial(0xc83929b868, 0xefebb8, 0x3, 0xc9cc6a2af7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:239 +0x512
net.Dial(0xefebb8, 0x3, 0xc9cc6a2af7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:193 +0x96
net/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc9cc6a2af7, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:665 +0x1d3
net/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc9cc6a2af0, 0x4, 0xc9cc6a2af7, 0x12, 0xc910b451a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:762 +0x1e3d
net/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc9cc6a2af0, 0x4, 0xc9cc6a2af7, 0x12, 0xca5810e1e0)
/usr/local/go/src/net/http/transport.go:706 +0x66
created by net/http.(*Transport).getConn
/usr/local/go/src/net/http/transport.go:708 +0x262
goroutine 330038 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954dc0, 0x4a817c800, 0x4a817c800, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:433 +0x6ea
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c
goroutine 332258 [chan receive]:
net/http.(*Transport).getConn.func2.1(0xc9387b2ba0, 0xc8f003a540, 0x1101a98)
/usr/local/go/src/net/http/transport.go:695 +0x48
created by net/http.(*Transport).getConn.func2
/usr/local/go/src/net/http/transport.go:699 +0x70
goroutine 296915 [chan send]:
github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter.func1(0xc820966f60, 0xc86e726ba0)
/go/src/github.com/prometheus/prometheus/storage/local/series.go:129 +0xe1
created by github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter
/go/src/github.com/prometheus/prometheus/storage/local/series.go:134 +0x5d
goroutine 298583 [chan send, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:207 +0x318
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 332040 [IO wait]:
net.runtime_pollWait(0x7f876d6fd0f8, 0x72, 0xca95c9c000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc959e18290, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc959e18290, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc959e18230, 0xca95c9c000, 0x1000, 0x1000, 0x0, 0x7f8775629050, 0xc82000e150)
/usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc886b48668, 0xca95c9c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
net/http.(*connReader).Read(0xca3e430500, 0xca95c9c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:526 +0x196
bufio.(*Reader).fill(0xc9307bdbc0)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc9307bdbc0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc9307bdbc0, 0x0, 0x0, 0x0, 0xeb8c00, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc8bf1fa390, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc8bf1fa390, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:36 +0x40
net/http.readRequest(0xc9307bdbc0, 0x700, 0xcb9da581c0, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:721 +0xb6
net/http.(*conn).readRequest(0xc9182a0380, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:705 +0x359
net/http.(*conn).serve(0xc9182a0380)
/usr/local/go/src/net/http/server.go:1425 +0x947
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2137 +0x44e
goroutine 331566 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7f876d6fc8b8, 0x77, 0x5c2e34)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc9cb03b560, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc9cb03b560, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc9cb03b500, 0x0, 0x0, 0x7f87753f37d0, 0xc94efd8d00, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:127 +0x28e
net.(*netFD).dial(0xc9cb03b500, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc923dcf620, 0x0, 0xc900000000, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:137 +0x364
net.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xc923dcf600, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc923dcf620, ...)
/usr/local/go/src/net/sock_posix.go:89 +0x429
net.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc923dcf620, 0x0, 0x0, 0x0, 0x1, ...)
/usr/local/go/src/net/ipsock_posix.go:161 +0x153
net.dialTCP(0xefebb8, 0x3, 0x0, 0xc923dcf620, 0x0, 0xc900000000, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b
net.dialSingle(0xc82130a2d0, 0x7f876d6fc448, 0xc923dcf620, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:371 +0x40c
net.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:343 +0x75
net.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xc923dcf620, 0xc94efd8ce0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc82130a2d0, 0xc95c4b4920, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:345 +0x7d0
net.(*Dialer).Dial(0xcb1c8f7868, 0xefebb8, 0x3, 0xc8e32dda97, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:239 +0x512
net.Dial(0xefebb8, 0x3, 0xc8e32dda97, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:193 +0x96
net/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc8e32dda97, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:665 +0x1d3
net/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc8e32dda90, 0x4, 0xc8e32dda97, 0x12, 0xc910b451a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:762 +0x1e3d
net/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc8e32dda90, 0x4, 0xc8e32dda97, 0x12, 0xc822bc2d80)
/usr/local/go/src/net/http/transport.go:706 +0x66
created by net/http.(*Transport).getConn
/usr/local/go/src/net/http/transport.go:708 +0x262
goroutine 330050 [select, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 298407 [chan send, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 329919 [select]:
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).sendLoop(0xc911dc4620, 0x7f87753f3770, 0xc941425100, 0xc8d6df11a0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:480 +0xd8c
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop.func1(0xc911dc4620, 0xc8d6df11a0, 0xc8ad730b20)
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:272 +0x48
created by github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:275 +0x60d
goroutine 329911 [select, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 297813 [chan send, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 331587 [chan receive, 1 minutes]:
net/http.(*Transport).getConn.func2.1(0xcb860ebce0, 0xc8f003a540, 0x1101a98)
/usr/local/go/src/net/http/transport.go:695 +0x48
created by net/http.(*Transport).getConn.func2
/usr/local/go/src/net/http/transport.go:699 +0x70
goroutine 330065 [select, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 329632 [chan receive, 3 minutes]:
github.com/prometheus/prometheus/retrieval/discovery.(*NerveDiscovery).processUpdates(0xc9793b6720)
created by github.com/prometheus/prometheus/retrieval/discovery.NewNerveDiscovery
goroutine 331395 [IO wait, 1 minutes]:
net.runtime_pollWait(0x7f876d6fcbb8, 0x77, 0x5c2e34)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc94643fd40, 0x77, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc94643fd40, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc94643fce0, 0x0, 0x0, 0x7f87753f37d0, 0xc9182e4fc0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:127 +0x28e
net.(*netFD).dial(0xc94643fce0, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc9a07054d0, 0x0, 0xc800000000, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/sock_posix.go:137 +0x364
net.socket(0xefebb8, 0x3, 0x2, 0x1, 0x0, 0xc9a0705400, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc9a07054d0, ...)
/usr/local/go/src/net/sock_posix.go:89 +0x429
net.internetSocket(0xefebb8, 0x3, 0x7f876d6fc4b0, 0x0, 0x7f876d6fc4b0, 0xc9a07054d0, 0x0, 0x0, 0x0, 0x1, ...)
/usr/local/go/src/net/ipsock_posix.go:161 +0x153
net.dialTCP(0xefebb8, 0x3, 0x0, 0xc9a07054d0, 0x0, 0xc800000000, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/tcpsock_posix.go:171 +0x12b
net.dialSingle(0xc9f3936bd0, 0x7f876d6fc448, 0xc9a07054d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:371 +0x40c
net.dialSerial.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:343 +0x75
net.dial(0xefebb8, 0x3, 0x7f876d6fc448, 0xc9a07054d0, 0xc9182e4fa0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc9f3936bd0, 0xc8e55da6e0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:345 +0x7d0
net.(*Dialer).Dial(0xc96dce7868, 0xefebb8, 0x3, 0xc9cae86697, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:239 +0x512
net.Dial(0xefebb8, 0x3, 0xc9cae86697, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/dial.go:193 +0x96
net/http.(*Transport).dial(0xc8f003a540, 0xefebb8, 0x3, 0xc9cae86697, 0x12, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:665 +0x1d3
net/http.(*Transport).dialConn(0xc8f003a540, 0x0, 0xc9cae86690, 0x4, 0xc9cae86697, 0x12, 0xc910b451a0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:762 +0x1e3d
net/http.(*Transport).getConn.func4(0xc8f003a540, 0x0, 0xc9cae86690, 0x4, 0xc9cae86697, 0x12, 0xcb860ebce0)
/usr/local/go/src/net/http/transport.go:706 +0x66
created by net/http.(*Transport).getConn
/usr/local/go/src/net/http/transport.go:708 +0x262
goroutine 329914 [chan receive, 3 minutes]:
github.com/prometheus/prometheus/retrieval/discovery.(*NerveDiscovery).Run(0xc9793b6720, 0x7f8775632620, 0xca2a8c3740, 0xca189398c0)
created by github.com/prometheus/prometheus/retrieval.(*targetSet).runProviders
/go/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:328 +0x308
goroutine 305585 [chan send, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:207 +0x318
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 305605 [chan send, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:207 +0x318
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 329631 [semacquire, 3 minutes]:
sync.runtime_Semacquire(0xc8ad730b2c)
/usr/local/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc8ad730b20)
/usr/local/go/src/sync/waitgroup.go:127 +0xb4
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.(*Conn).loop(0xc911dc4620)
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:287 +0x675
github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.ConnectWithDialer.func1(0xc911dc4620)
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:185 +0x21
created by github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk.ConnectWithDialer
/go/src/github.com/prometheus/prometheus/vendor/github.com/samuel/go-zookeeper/zk/conn.go:189 +0x5b1
goroutine 331663 [chan receive]:
net/http.(*Transport).getConn.func2.1(0xc9602e5e60, 0xc8f003a540, 0x1101a98)
/usr/local/go/src/net/http/transport.go:695 +0x48
created by net/http.(*Transport).getConn.func2
/usr/local/go/src/net/http/transport.go:699 +0x70
goroutine 331664 [IO wait]:
net.runtime_pollWait(0x7f876d6fc738, 0x72, 0xc8faf54000)
/usr/local/go/src/runtime/netpoll.go:160 +0x60
net.(*pollDesc).Wait(0xc8ec7a8c30, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8ec7a8c30, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8ec7a8bd0, 0xc8faf54000, 0x1000, 0x1000, 0x0, 0x7f8775629050, 0xc82000e150)
/usr/local/go/src/net/fd_unix.go:250 +0x23a
net.(*conn).Read(0xc929cabff0, 0xc8faf54000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:172 +0xe4
net/http.(*connReader).Read(0xca3a9ee500, 0xc8faf54000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:526 +0x196
bufio.(*Reader).fill(0xc936a17d40)
/usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).ReadSlice(0xc936a17d40, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:328 +0x21a
bufio.(*Reader).ReadLine(0xc936a17d40, 0x0, 0x0, 0x0, 0xeb8c00, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:357 +0x53
net/textproto.(*Reader).readLineSlice(0xc8bbdd61e0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:55 +0x81
net/textproto.(*Reader).ReadLine(0xc8bbdd61e0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/textproto/reader.go:36 +0x40
net/http.readRequest(0xc936a17d40, 0x900, 0xc9a519e7e0, 0x0, 0x0)
/usr/local/go/src/net/http/request.go:721 +0xb6
net/http.(*conn).readRequest(0xc976b0d900, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:705 +0x359
net/http.(*conn).serve(0xc976b0d900)
/usr/local/go/src/net/http/server.go:1425 +0x947
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2137 +0x44e
goroutine 329910 [select, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 329907 [select]:
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc8f1954d20, 0x4a817c800, 0x4a817c800, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:433 +0x6ea
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c
goroutine 330066 [select, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 305620 [chan send, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:205 +0x1f8
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 331254 [chan receive, 1 minutes]:
net/http.(*Transport).getConn.func2.1(0xc956e44b40, 0xc8f003a540, 0x1101a98)
/usr/local/go/src/net/http/transport.go:695 +0x48
created by net/http.(*Transport).getConn.func2
/usr/local/go/src/net/http/transport.go:699 +0x70
goroutine 332214 [select]:
net/http.(*Transport).getConn(0xc8f003a540, 0xc97b21d260, 0x0, 0xc9cc6a2af0, 0x4, 0xc9cc6a2af7, 0x12, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:711 +0x4ef
net/http.(*Transport).RoundTrip(0xc8f003a540, 0xc97b21d260, 0xc8f003a540, 0x0, 0x0)
/usr/local/go/src/net/http/transport.go:311 +0x7e9
net/http.send(0xc97b21d260, 0x7f8775631648, 0xc8f003a540, 0x0, 0x0, 0x0, 0xc9cc6a2b40, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:260 +0x6b7
net/http.(*Client).send(0xc910b451a0, 0xc97b21d260, 0x0, 0x0, 0x0, 0xc86a9a7000, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:155 +0x185
net/http.(*Client).doFollowingRedirects(0xc910b451a0, 0xc97b21d260, 0x1101ab8, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:475 +0x8a4
net/http.(*Client).Do(0xc910b451a0, 0xc97b21d260, 0x1102020, 0x0, 0x0)
/usr/local/go/src/net/http/client.go:188 +0xff
github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp.Do.func1(0xc910b451a0, 0xc97b21d260, 0x1101420, 0xca5810e180)
/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:49 +0x2f
created by github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp.Do
/go/src/github.com/prometheus/prometheus/vendor/golang.org/x/net/context/ctxhttp/ctxhttp.go:52 +0x10d
goroutine 330049 [select, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:203 +0x360
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:210 +0x789
goroutine 331875 [chan receive, 1 minutes]:
net/http.(*Transport).getConn.func2.1(0xc822bc2d80, 0xc8f003a540, 0x1101a98)
/usr/local/go/src/net/http/transport.go:695 +0x48
created by net/http.(*Transport).getConn.func2
/usr/local/go/src/net/http/transport.go:699 +0x70
goroutine 329912 [select, 3 minutes]:
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:94 +0xd85
/go/src/github.com/prometheus/prometheus/util/treecache/treecache.go:72 +0x32e

@brian-brazil brian-brazil added the kind/bug label Jul 4, 2016

@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jul 4, 2016

Thanks for reporting! Does it always crash with the same error (concurrent map read and map write) or different ones?

This seems very odd. Each query gets its own copy of the Metric of a given sample during query analysis time, and there is no concurrency within a query that could cause concurrent read/write accesses to it. Also, there seems to be only one query running anyways (judging by the trace).
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Jul 4, 2016

Does it always crash in model.SignatureForLabels?
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jul 4, 2016 •
edited

The only source to me seems that we modify the underlying model.Metric we got from the storage in the storage simultaneously.
But to my awareness this is not happening anywhere.
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jul 4, 2016

Right - that's why I'm suspecting either general memory corruption or a Go bug. If the error message is always the same, it's unlikely to be the former, but could still be the latter.
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jul 4, 2016 •
edited

FWIW, I cannot imagine this being a Go bug. Maps are heavily used and the panic on unlocked concurrent access has been around for a while now.
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jul 4, 2016

Yeah, that'd be really unlikely too. Let's see what @prfalken says about the error messages.
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jul 4, 2016

But we only do a copy of each metric on returning the result or if we have to modify the metric during querying. Otherwise during querying we directly reference the map from the storage if I recall/checked correctly.
Still no concurrency in the query engine where the write could have happened.
@prfalken

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

prfalken commented Jul 4, 2016

I'm digging in logs. Crash recoveries on 1.2M metrics makes them difficult to find :)
Keep you posted.
@prfalken

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

prfalken commented Jul 4, 2016

Another crash today, totally different trace :

time="2016-07-04T14:08:02Z" level=warning msg="Error on ingesting out-of-order samples" numDropped=1 source="scrape.go:467"
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x5 pc=0x56a6e3]
goroutine 504 [running]:
panic(0xd864c0, 0xc820010090)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/prometheus/prometheus/storage/local.doubleDeltaEncodedChunk.add(0x0, 0xdeaddeaddeaddead, 0x400, 0x155b63c9338, 0x3fe8618618618618, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/doubledelta.go:86 +0x1a3
github.com/prometheus/prometheus/storage/local.(*doubleDeltaEncodedChunk).add(0xc97982dd00, 0x155b63c9338, 0x3fe8618618618618, 0x0, 0x0, 0x0, 0x0, 0x0)
<autogenerated>:33 +0xdd
github.com/prometheus/prometheus/storage/local.(*chunkDesc).add(0xc901b0b480, 0x155b63c9338, 0x3fe8618618618618, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/chunk.go:123 +0x7a
github.com/prometheus/prometheus/storage/local.(*memorySeries).add(0xc89329fab0, 0x155b63c9338, 0x3fe8618618618618, 0xc89329fab0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/series.go:241 +0x296
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Append(0xc820330000, 0xc9db5f0700, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:656 +0x4ed
github.com/prometheus/prometheus/storage.Fanout.Append(0xc8202e2bd0, 0x1, 0x1, 0xc9db5f0700, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/storage.go:60 +0xc8
github.com/prometheus/prometheus/storage.(*Fanout).Append(0xc8202e0cc0, 0xc9db5f0700, 0x0, 0x0)
<autogenerated>:3 +0xb7
github.com/prometheus/prometheus/retrieval.honorLabelsAppender.Append(0x7f1db4e01750, 0xc8202e0cc0, 0xc9d5271830, 0xc9db5f0700, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/target.go:264 +0x1d0
github.com/prometheus/prometheus/retrieval.(*honorLabelsAppender).Append(0xc9958b0040, 0xc9db5f0700, 0x0, 0x0)
<autogenerated>:30 +0xb7
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).append(0xc957163270, 0xcaa8270000, 0x16cab, 0x17800)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:453 +0xc7
github.com/prometheus/prometheus/retrieval.(*scrapeLoop).run(0xc957163270, 0x4a817c800, 0x4a817c800, 0x0)
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:422 +0x539
created by github.com/prometheus/prometheus/retrieval.(*scrapePool).reload.func1
/go/src/github.com/prometheus/prometheus/retrieval/scrape.go:196 +0x7c
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jul 5, 2016

@beorn7 ^
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jul 5, 2016

In theory, this could be a result of a crash recovery gone wild.

@prfalken could you check more crashes? If each shows you a different trace, the odds are you are having a corrupt binary, broken hardware, or wildly corrupted storage data. (Among those, arguably the latter should not cause crashes.)

@beorn7 beorn7 referenced this issue Jul 5, 2016

Closed

storage: memorySeries.headChunkClosed not set correctly? #1790

@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jul 5, 2016

I'll filed #1790 to investigate the assumed crash recovery issue.
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jul 6, 2016

Yeah, have you tried running the same Prometheus server on different hardware? And then, also try the latest version, 0.20.0.
@prfalken

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

prfalken commented Jul 6, 2016

New crash today :

panic: interface conversion: interface {} is nil, not *local.chunkDesc
goroutine 245 [running]:
panic(0xdaa100, 0xca4ec98300)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).maybeEvict(0xc820255900)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:872 +0x2b0
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleEvictList(0xc820255900)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:831 +0x1cb
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:353 +0x4b8

I'm going to upgrade to 0.20.
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jul 6, 2016

These are 3 very different crashes and none of them have been observed anywhere before, even on high load servers. I wouldn't be surprised if something outside of our code and the Go compiler is broken here.
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jul 7, 2016 •
edited

I found a plausible reason for the 2nd stack trace reported above. See #1798.
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jul 7, 2016

So the first crash could have caused the 2nd one via broken crash recovery. Still doesn't explain the 3rd.
@prfalken

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

prfalken commented Jul 8, 2016

Now prometheus finishes its crash recovery after more than half an hour and crashes again over and over :

time="2016-07-08T15:36:18Z" level=info msg="14190000 metrics queued for indexing." source="crashrecovery.go:517"
time="2016-07-08T15:36:18Z" level=info msg="All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)" source="crashrecovery.go:523"
time="2016-07-08T15:36:18Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1355"
time="2016-07-08T15:36:18Z" level=info msg="Done checkpointing fingerprint mappings in 2.518278ms." source="persistence.go:1378"
time="2016-07-08T15:36:18Z" level=warning msg="Crash recovery complete." source="crashrecovery.go:150"
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
panic(0xd864c0, 0xc820010050)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/prometheus/prometheus/storage/local/codable.getBuf(0xffd741a0a4e77f01, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/codable/codable.go:64 +0x170
github.com/prometheus/prometheus/storage/local/codable.decodeString(0x7fbf6e67e920, 0xc820558060, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/codable/codable.go:142 +0xd4
github.com/prometheus/prometheus/storage/local/codable.(*Metric).UnmarshalFromReader(0xc82c73f858, 0x7fbf6e67e920, 0xc820558060, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/codable/codable.go:193 +0x195
github.com/prometheus/prometheus/storage/local.(*headsScanner).scan(0xc82000e190, 0xc821d34060)
/go/src/github.com/prometheus/prometheus/storage/local/heads.go:122 +0x1fe
github.com/prometheus/prometheus/storage/local.(*persistence).loadSeriesMapAndHeads(0xc820212b00, 0xc820600040, 0x0, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/persistence.go:712 +0x1c0
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start(0xc82025b900, 0x0, 0x0)
/go/src/github.com/prometheus/prometheus/storage/local/storage.go:341 +0x275
main.Main(0x0)
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:142 +0x12ac
main.main()
/go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:41 +0x18

Is there any way to fix this without trashing the whole data directory ?
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jul 8, 2016

It looks your data is corrupted beyond hope. The above can only happen if the string length read from the checkpoint is negative. Arguably, we should not crash in that case. (I filed #1800 about it.) However, with this kind of corruption, there is little hope anything useful will come out of a crash-free decoding. (As well as giving you a negative number for the string length, it could have given you a string length of one trillion or something, and you would simply run out of memory. Also, after reading a wrong string length, all the other data is misaligned, and you will essentially read noise from the checkpoint.)
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Aug 8, 2016

@prfalken did this problem reoccur with a server that was not most likely suffering from data corruption?
@prfalken

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

prfalken commented Aug 16, 2016 •
edited

Ran a new prometheus server on older hardware, other manufacturer. Prometheus works like a charm and never crashes. Looks like our Dell servers have hardware issues, and prometheus is not the only one to crash. How disappointing.

Sorry for wasting your time. Hope I could help to discover new errors to catch.
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Aug 16, 2016

Thanks! Sorry about your servers...

@juliusv juliusv closed this Aug 16, 2016

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.