Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rule evaluation starting prematurely #1297

Closed
RichiH opened this Issue Jan 8, 2016 · 11 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@RichiH @beorn7 @juliusv @fabxc
@RichiH
Copy link
Member

RichiH commented Jan 8, 2016 

foo@prometheus % ./prometheus -alertmanager.url http://$URL:9093
prometheus, version 0.16.1 (branch: master, revision: adb3be4)
  build user:       foo
  build date:       20160107-14:51:02
  go version:       1.5.1
INFO[0000] Loading configuration file prometheus.yml     source=main.go:195
INFO[0000] Loading series map and head chunks...         source=storage.go:262
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x578778]

goroutine 33 [running]:
github.com/prometheus/prometheus/storage/local.(*seriesMap).get(0x0, 0x191ef2fcc7f825c1, 0x0, 0x191ef2fcc7f82500)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/series.go:68 +0xd8
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).MetricForFingerprint(0xc8201f80f0, 0x191ef2fcc7f825c1, 0xc83b3cb100, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:493 +0xd5
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).MetricsForLabelMatchers(0xc8201f80f0, 0xc8201093c8, 0x1, 0x1, 0x1)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:467 +0x304
github.com/prometheus/prometheus/promql.(*Analyzer).Analyze.func2(0x7fda44271718, 0xc83b39c0c0, 0x10)
        /home/gocode/src/github.com/prometheus/prometheus/promql/analyzer.go:74 +0x3e5
github.com/prometheus/prometheus/promql.inspector.Visit(0xc83b3aa020, 0x7fda44271718, 0xc83b39c0c0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/ast.go:303 +0x40
github.com/prometheus/prometheus/promql.Walk(0x7fda453174e8, 0xc83b3aa020, 0x7fda44271718, 0xc83b39c0c0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/ast.go:252 +0x65
github.com/prometheus/prometheus/promql.Walk(0x7fda453174e8, 0xc83b3aa020, 0x7fda442716c0, 0xc83b39c140)
        /home/gocode/src/github.com/prometheus/prometheus/promql/ast.go:278 +0x881
github.com/prometheus/prometheus/promql.Inspect(0x7fda442716c0, 0xc83b39c140, 0xc83b3aa020)
        /home/gocode/src/github.com/prometheus/prometheus/promql/ast.go:313 +0x60
github.com/prometheus/prometheus/promql.(*Analyzer).Analyze(0xc83b39c240, 0x7fda45317480, 0xc83b3b0120, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/analyzer.go:102 +0x15e
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc8201e3340, 0x7fda45317480, 0xc83b3b0120, 0xc83b39c180, 0xc83b3be060, 0x0, 0x0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/engine.go:377 +0x276
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc8201e3340, 0xc83b39c180, 0x0, 0x0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/engine.go:357 +0x4ca
github.com/prometheus/prometheus/promql.(*query).Exec(0xc83b39c180, 0xc83b398110)
        /home/gocode/src/github.com/prometheus/prometheus/promql/engine.go:195 +0x2e
github.com/prometheus/prometheus/rules.(*AlertingRule).eval(0xc82016aa00, 0x15221d2c77b, 0xc8201e3340, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/rules/alerting.go:146 +0x140
github.com/prometheus/prometheus/rules.(*Group).eval.func1(0xc83b398000, 0x15221d2c77b, 0xc82016a870, 0x7fda44271808, 0xc82016aa00)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:217 +0xef
created by github.com/prometheus/prometheus/rules.(*Group).eval
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:243 +0x128

goroutine 1 [runnable]:
github.com/prometheus/prometheus/storage/local/codable.(*Metric).UnmarshalFromReader(0xc8305c37c8, 0x7fda44230050, 0xc8200e8120, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/codable/codable.go:197 +0x239
github.com/prometheus/prometheus/storage/local.(*persistence).loadSeriesMapAndHeads(0xc820424500, 0xc820170180, 0x28332, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:765 +0x108d
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start(0xc8201f80f0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:263 +0x25d
main.Main(0x0)
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:136 +0xe0e
main.main()
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:46 +0x18

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 39 [syscall]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 26 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820118580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 120 [select, locked to thread]:
runtime.gopark(0xfb5e98, 0xc8205b5728, 0xde9db8, 0x6, 0x434218, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc8205b5728, 0x0, 0x18)
        /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc8205b5728)
        /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 119 [runnable]:
sync.runtime_Semacquire(0xc83b39800c)
        /usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*WaitGroup).Wait(0xc83b398000)
        /usr/local/go/src/sync/waitgroup.go:126 +0xb4
github.com/prometheus/prometheus/rules.(*Group).eval(0xc82016a870)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:245 +0x155
github.com/prometheus/prometheus/rules.(*Group).run.func1()
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:130 +0x4e
github.com/prometheus/prometheus/rules.(*Group).run(0xc82016a870)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:134 +0xa1
created by github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:401 +0x96

goroutine 121 [chan receive]:
main.Main.func2(0xc8201f73e0, 0xc8201f7380, 0xc8201fc200, 0xc8200eecd0, 0x5, 0x5)
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:125 +0x45
created by main.Main
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:133 +0xdf4

goroutine 122 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201fa7e0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 123 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820118b00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 124 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820118b00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 125 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820118b00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 126 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820118b00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 127 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820118b00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 128 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201fa2a0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 27 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820118580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 28 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820118580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 29 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820118580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 30 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820118580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 31 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201fa540)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 129 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820118840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 130 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820118840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 131 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820118840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 132 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820118840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 133 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820118840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 134 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201faa80)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 146 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc820118c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 147 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc820118c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 148 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc820118c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 149 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc820118c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 150 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc820118c60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 151 [select]:
github.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc820424500)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:1341 +0xcaf
github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc820424500)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:280 +0x21
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:252 +0x137
@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Jan 8, 2016 

% cat prometheus.rules
job_service:rpc_durations_microseconds_count:avg_rate5m = avg(rate(rpc_durations_microseconds_count[5m])) by (job, service)

ALERT InstanceDown
  IF up == 0
  FOR 5m
  LABELS { severity = "page" }
  ANNOTATIONS {
    summary = "Instance {{ $labels.instance }} down",
    description = "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.",
  }
%
@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Jan 8, 2016 

~/gocode/src/github.com/prometheus/alertmanager % git log | head -n1
commit 3de6b062a29ca874e0e4c4c137b8acb5d9887627
~/gocode/src/github.com/prometheus/alertmanager %
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jan 8, 2016

Looks like the server has started evaluating expressions before the series map is loaded from disk. @fabxc might know most about it.

@RichiH The commit you have provided is for prometheus/alertmanager, but the crash is in prometheus/prometheus.

@beorn7 beorn7 added bug labels Jan 8, 2016

@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Jan 8, 2016

@beorn7 I know, I just wanted to let you know that I was using the new AM, not the old-style.
@beorn7

This comment has been minimized.

Sign in to view
Copy link
Member

beorn7 commented Jan 8, 2016

Can you let us know the commit for prometheus/prometheus, too?
@juliusv

This comment has been minimized.

Sign in to view
Copy link
Member

juliusv commented Jan 8, 2016

Says at the top: adb3be4 (i.e. 0.16.1)
@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Jan 8, 2016

It's in the startup, but sure:

~/gocode/src/github.com/prometheus/prometheus % git log | head -n1
commit adb3be4923d1ef48c1532dd515c9dfeae6419e32
 ~/gocode/src/github.com/prometheus/prometheus %
@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Jan 8, 2016

During one of the recoveries while reproducing that this was caused by passing -alertmanager.url, I also ran into:

% ./prometheus
prometheus, version 0.16.1 (branch: master, revision: adb3be4)
  build user:       foo
  build date:       20160107-14:51:02
  go version:       1.5.1
INFO[0000] Loading configuration file prometheus.yml     source=main.go:195
INFO[0000] Loading series map and head chunks...         source=storage.go:262
WARN[0004] Persistence layer appears dirty.              source=persistence.go:699
WARN[0004] Starting crash recovery. Prometheus is inoperational until complete.  source=crashrecovery.go:38
WARN[0004] To avoid crash recovery in the future, shut down Prometheus with SIGTERM or a HTTP POST to /-/quit.  source=crashrecovery.go:39
INFO[0004] Scanning files.                               source=crashrecovery.go:53
INFO[0004] 10000 files scanned.                          source=crashrecovery.go:75
INFO[0004] 20000 files scanned.                          source=crashrecovery.go:75
INFO[0004] 30000 files scanned.                          source=crashrecovery.go:75
INFO[0005] 40000 files scanned.                          source=crashrecovery.go:75
INFO[0005] 50000 files scanned.                          source=crashrecovery.go:75
INFO[0005] 60000 files scanned.                          source=crashrecovery.go:75
INFO[0005] 70000 files scanned.                          source=crashrecovery.go:75
INFO[0005] 80000 files scanned.                          source=crashrecovery.go:75
INFO[0006] 90000 files scanned.                          source=crashrecovery.go:75
INFO[0006] 100000 files scanned.                         source=crashrecovery.go:75
INFO[0006] 110000 files scanned.                         source=crashrecovery.go:75
INFO[0006] 120000 files scanned.                         source=crashrecovery.go:75
INFO[0006] 130000 files scanned.                         source=crashrecovery.go:75
INFO[0007] 140000 files scanned.                         source=crashrecovery.go:75
INFO[0007] 150000 files scanned.                         source=crashrecovery.go:75
INFO[0007] 160000 files scanned.                         source=crashrecovery.go:75
INFO[0007] 170000 files scanned.                         source=crashrecovery.go:75
INFO[0008] 180000 files scanned.                         source=crashrecovery.go:75
INFO[0008] 190000 files scanned.                         source=crashrecovery.go:75
INFO[0008] 200000 files scanned.                         source=crashrecovery.go:75
INFO[0008] 210000 files scanned.                         source=crashrecovery.go:75
INFO[0009] 220000 files scanned.                         source=crashrecovery.go:75
INFO[0009] 230000 files scanned.                         source=crashrecovery.go:75
INFO[0009] 240000 files scanned.                         source=crashrecovery.go:75
INFO[0009] 250000 files scanned.                         source=crashrecovery.go:75
INFO[0009] 260000 files scanned.                         source=crashrecovery.go:75
INFO[0009] 270000 files scanned.                         source=crashrecovery.go:75
INFO[0010] 280000 files scanned.                         source=crashrecovery.go:75
INFO[0010] 290000 files scanned.                         source=crashrecovery.go:75
INFO[0010] 300000 files scanned.                         source=crashrecovery.go:75
INFO[0010] 310000 files scanned.                         source=crashrecovery.go:75
INFO[0010] File scan complete. 318522 series found.      source=crashrecovery.go:80
INFO[0010] Checking for series without series file.      source=crashrecovery.go:82
INFO[0010] Check for series without series file complete.  source=crashrecovery.go:127
INFO[0010] Cleaning up archive indexes.                  source=crashrecovery.go:363
INFO[0011] 10000 archived metrics checked.               source=crashrecovery.go:370
INFO[0011] 20000 archived metrics checked.               source=crashrecovery.go:370
INFO[0011] 30000 archived metrics checked.               source=crashrecovery.go:370
INFO[0011] 40000 archived metrics checked.               source=crashrecovery.go:370
INFO[0012] 50000 archived metrics checked.               source=crashrecovery.go:370
INFO[0012] 60000 archived metrics checked.               source=crashrecovery.go:370
INFO[0012] 70000 archived metrics checked.               source=crashrecovery.go:370
INFO[0012] 80000 archived metrics checked.               source=crashrecovery.go:370
INFO[0013] 90000 archived metrics checked.               source=crashrecovery.go:370
INFO[0013] 100000 archived metrics checked.              source=crashrecovery.go:370
INFO[0013] 110000 archived metrics checked.              source=crashrecovery.go:370
INFO[0013] 120000 archived metrics checked.              source=crashrecovery.go:370
INFO[0013] 130000 archived metrics checked.              source=crashrecovery.go:370
INFO[0014] 140000 archived metrics checked.              source=crashrecovery.go:370
INFO[0014] 150000 archived metrics checked.              source=crashrecovery.go:370
INFO[0014] 160000 archived metrics checked.              source=crashrecovery.go:370
INFO[0015] 170000 archived metrics checked.              source=crashrecovery.go:370
INFO[0015] 180000 archived metrics checked.              source=crashrecovery.go:370
INFO[0015] 10000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0015] 20000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0015] 30000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0016] 40000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0016] 50000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0016] 60000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0016] 70000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0016] 80000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0016] 90000 archived time ranges checked.           source=crashrecovery.go:427
INFO[0016] 100000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0017] 110000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0017] 120000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0017] 130000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0017] 140000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0017] 150000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0017] 160000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0017] 170000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0017] 180000 archived time ranges checked.          source=crashrecovery.go:427
INFO[0018] Clean-up of archive indexes complete.         source=crashrecovery.go:451
INFO[0018] Rebuilding label indexes.                     source=crashrecovery.go:459
INFO[0018] Indexing metrics in memory.                   source=crashrecovery.go:460
INFO[0018] 10000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 20000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 30000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 40000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 50000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 60000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 70000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 80000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 90000 metrics queued for indexing.            source=crashrecovery.go:465
INFO[0018] 100000 metrics queued for indexing.           source=crashrecovery.go:465
INFO[0018] 110000 metrics queued for indexing.           source=crashrecovery.go:465
INFO[0018] 120000 metrics queued for indexing.           source=crashrecovery.go:465
INFO[0018] Indexing archived metrics.                    source=crashrecovery.go:468
INFO[0018] 130000 metrics queued for indexing.           source=crashrecovery.go:481
INFO[0018] 140000 metrics queued for indexing.           source=crashrecovery.go:481
INFO[0018] 150000 metrics queued for indexing.           source=crashrecovery.go:481
INFO[0018] 160000 metrics queued for indexing.           source=crashrecovery.go:481
INFO[0018] 170000 metrics queued for indexing.           source=crashrecovery.go:481
INFO[0019] 180000 metrics queued for indexing.           source=crashrecovery.go:481
INFO[0019] 190000 metrics queued for indexing.           source=crashrecovery.go:481
INFO[0019] 200000 metrics queued for indexing.           source=crashrecovery.go:481
INFO[0019] 210000 metrics queued for indexing.           source=crashrecovery.go:481
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x578778]

goroutine 156 [running]:
github.com/prometheus/prometheus/storage/local.(*seriesMap).get(0x0, 0xe608045c4c1be2ab, 0x0, 0xe608045c4c1be200)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/series.go:68 +0xd8
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).MetricForFingerprint(0xc82016e0f0, 0xe608045c4c1be2ab, 0xc843575100, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:493 +0xd5
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).MetricsForLabelMatchers(0xc82016e0f0, 0xc8607abdf0, 0x1, 0x1, 0x1)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:467 +0x304
github.com/prometheus/prometheus/promql.(*Analyzer).Analyze.func2(0x7feab00697a8, 0xc85d06dac0, 0x10)
        /home/gocode/src/github.com/prometheus/prometheus/promql/analyzer.go:74 +0x3e5
github.com/prometheus/prometheus/promql.inspector.Visit(0xc8621f65c0, 0x7feab00697a8, 0xc85d06dac0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/ast.go:303 +0x40
github.com/prometheus/prometheus/promql.Walk(0x7feaa14b0368, 0xc8621f65c0, 0x7feab00697a8, 0xc85d06dac0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/ast.go:252 +0x65
github.com/prometheus/prometheus/promql.Walk(0x7feaa14b0368, 0xc8621f65c0, 0x7feab0069750, 0xc85d06db40)
        /home/gocode/src/github.com/prometheus/prometheus/promql/ast.go:278 +0x881
github.com/prometheus/prometheus/promql.Inspect(0x7feab0069750, 0xc85d06db40, 0xc8621f65c0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/ast.go:313 +0x60
github.com/prometheus/prometheus/promql.(*Analyzer).Analyze(0xc85d06dc00, 0x7feaa14b0300, 0xc85d4bbe60, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/analyzer.go:102 +0x15e
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc820224080, 0x7feaa14b0300, 0xc85d4bbe60, 0xc85d06db80, 0xc8446ede30, 0x0, 0x0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/engine.go:377 +0x276
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc820224080, 0xc85d06db80, 0x0, 0x0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/promql/engine.go:357 +0x4ca
github.com/prometheus/prometheus/promql.(*query).Exec(0xc85d06db80, 0xc847cc1130)
        /home/gocode/src/github.com/prometheus/prometheus/promql/engine.go:195 +0x2e
github.com/prometheus/prometheus/rules.(*AlertingRule).eval(0xc820120960, 0x15221d33cac, 0xc820224080, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/rules/alerting.go:146 +0x140
github.com/prometheus/prometheus/rules.(*Group).eval.func1(0xc847cdc060, 0x15221d33cac, 0xc8201207d0, 0x7feab0069898, 0xc820120960)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:217 +0xef
created by github.com/prometheus/prometheus/rules.(*Group).eval
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:243 +0x128

goroutine 1 [runnable]:
syscall.Syscall(0x1, 0x2, 0xc863c88370, 0x67, 0x67, 0x67, 0x0)
        /usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.write(0x2, 0xc863c88370, 0x67, 0xa7, 0xc861a2e850, 0x0, 0x0)
        /usr/local/go/src/syscall/zsyscall_linux_amd64.go:1064 +0x5f
syscall.Write(0x2, 0xc863c88370, 0x67, 0xa7, 0x0, 0x0, 0x0)
        /usr/local/go/src/syscall/syscall_unix.go:176 +0x4d
os.(*File).write(0xc82002e018, 0xc863c88370, 0x67, 0xa7, 0x0, 0x0, 0x0)
        /usr/local/go/src/os/file_unix.go:232 +0xaa
os.(*File).Write(0xc82002e018, 0xc863c88370, 0x67, 0xa7, 0xd17d40, 0x0, 0x0)
        /usr/local/go/src/os/file.go:139 +0x8a
bytes.(*Buffer).WriteTo(0xc84838bf80, 0x7feab3774330, 0xc82002e018, 0x0, 0x0, 0x0)
        /usr/local/go/src/bytes/buffer.go:206 +0xc8
io.copyBuffer(0x7feab3774330, 0xc82002e018, 0x7feab112b2d8, 0xc84838bf80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/io.go:371 +0xd0
io.Copy(0x7feab3774330, 0xc82002e018, 0x7feab112b2d8, 0xc84838bf80, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/io.go:351 +0x64
github.com/prometheus/prometheus/vendor/github.com/Sirupsen/logrus.Entry.log(0xc8200142c0, 0xc861a28b40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x412e04, 0xc861a28ba0, ...)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/Sirupsen/logrus/entry.go:104 +0x4c0
github.com/prometheus/prometheus/vendor/github.com/Sirupsen/logrus.(*Entry).Info(0xc8626b4540, 0xc843578f08, 0x1, 0x1)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/Sirupsen/logrus/entry.go:129 +0x8c
github.com/prometheus/prometheus/vendor/github.com/Sirupsen/logrus.(*Entry).Infof(0xc8626b4540, 0xe9f1e0, 0x1f, 0xc843579020, 0x1, 0x1)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/Sirupsen/logrus/entry.go:173 +0x139
github.com/prometheus/prometheus/vendor/github.com/prometheus/common/log.Infof(0xe9f1e0, 0x1f, 0xc843579020, 0x1, 0x1)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/common/log/log.go:258 +0x63
github.com/prometheus/prometheus/storage/local.(*persistence).rebuildLabelIndexes.func1(0x7feaa155dc88, 0xc82fb0e4e0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/crashrecovery.go:481 +0x2f4
github.com/prometheus/prometheus/storage/local/index.(*LevelDB).ForEach(0xc8202d6360, 0xc864688240, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/index/leveldb.go:158 +0x1d1
github.com/prometheus/prometheus/storage/local.(*persistence).rebuildLabelIndexes(0xc8203dc700, 0xc8205e5b60, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/crashrecovery.go:471 +0x637
github.com/prometheus/prometheus/storage/local.(*persistence).recoverFromCrash(0xc8203dc700, 0xc8205e5b60, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/crashrecovery.go:132 +0xf6e
github.com/prometheus/prometheus/storage/local.(*persistence).loadSeriesMapAndHeads.func1(0xc843579a60, 0xc8203dc700, 0xc843579a70, 0xc8205e5b60, 0xc843579780)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:700 +0x12b
github.com/prometheus/prometheus/storage/local.(*persistence).loadSeriesMapAndHeads(0xc8203dc700, 0xc8205e6660, 0x34f4f, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:866 +0x250d
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start(0xc82016e0f0, 0x0, 0x0)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:263 +0x25d
main.Main(0x0)
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:136 +0xe0e
main.main()
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:46 +0x18

goroutine 17 [syscall, 5 minutes, locked to thread]:
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 37 [syscall, 5 minutes]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 11 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc82014ab00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 133 [select, 5 minutes, locked to thread]:
runtime.gopark(0xfb5e98, 0xc82002cf28, 0xde9db8, 0x6, 0x434218, 0x2)
        /usr/local/go/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc82002cf28, 0x0, 0x18)
        /usr/local/go/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82002cf28)
        /usr/local/go/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal1_unix.go:227 +0x353
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 132 [runnable]:
sync.runtime_Semacquire(0xc847cdc06c)
        /usr/local/go/src/runtime/sema.go:43 +0x26
sync.(*WaitGroup).Wait(0xc847cdc060)
        /usr/local/go/src/sync/waitgroup.go:126 +0xb4
github.com/prometheus/prometheus/rules.(*Group).eval(0xc8201207d0)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:245 +0x155
github.com/prometheus/prometheus/rules.(*Group).run.func1()
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:130 +0x4e
github.com/prometheus/prometheus/rules.(*Group).run(0xc8201207d0)
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:148 +0x1e3
created by github.com/prometheus/prometheus/rules.(*Manager).ApplyConfig.func1
        /home/gocode/src/github.com/prometheus/prometheus/rules/manager.go:401 +0x96

goroutine 134 [chan receive, 5 minutes]:
main.Main.func2(0xc820169260, 0xc820169200, 0xc820142480, 0xc8200f3540, 0x5, 0x5)
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:125 +0x45
created by main.Main
        /home/gocode/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:133 +0xdf4

goroutine 135 [runnable]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc82014c9a0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 12 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc82014ab00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 13 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc82014ab00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 14 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc82014ab00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 15 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc82014ab00)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 16 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc82014c380)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 97 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc82014a580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 146 [select, 5 minutes]:

github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc82014a580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 147 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc82014a580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 148 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc82014a580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 149 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc82014a580)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 150 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc820102380)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 136 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc82014a840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 137 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc82014a840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 138 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc82014a840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 139 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc82014a840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 140 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc82014a840)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 141 [select]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201021c0)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29d
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x26b

goroutine 142 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc82014ac60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x54a
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x77f

goroutine 143 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc82014ac60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14b
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7a1

goroutine 144 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc82014ac60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x7c8
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9a5

goroutine 145 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc82014ac60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x253
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9c7

goroutine 162 [select, 5 minutes]:
github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc82014ac60)
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x1a2
created by github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb.openDB
        /home/gocode/src/github.com/prometheus/prometheus/vendor/github.com/syndtr/goleveldb/leveldb/db.go:148 +0x9e9

goroutine 163 [runnable]:
github.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc8203dc700)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:1341 +0xcaf
github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc8203dc700)
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/persistence.go:280 +0x21
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
        /home/gocode/src/github.com/prometheus/prometheus/storage/local/storage.go:252 +0x137
%
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jan 8, 2016

This is related to rule evaluation starting before the storage is ready. Not related to the Alertmanager URL.

@fabxc fabxc changed the title Crash bug: `./prometheus -alertmanager.url http://URL:9093` Rule evaluation starting prematurely Jan 8, 2016

@fabxc fabxc referenced this issue Jan 8, 2016

Merged

Fix premature rule evaluation #1298

@RichiH

This comment has been minimized.

Sign in to view
Copy link
Member Author

RichiH commented Jan 8, 2016

@fabxc I am torn between two possible replies:

Pun 1
s/v/j/
Pun 2

This kills the bug

Either way, I can't reproduce the crash, now.

@fabxc fabxc closed this in #1298 Jan 8, 2016

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 24, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 24, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.