storage.local.retention killing prometheus?

[mokshpooja]

Hi Guys,
I have 2 setups, one on my local development setup (vagrant+vm+docker) and the other aws+docker. In both setups I have prometheus+cAdvisor+Alertmanager running as seperate containers.
I want to clean the disk storage every 15 minutes so I added below in my prometheus dockerfile-
CMD "-storage.local.retention=15m"
It seems like after 15 mins my prometheus container dies.
My aim is to clean the collected metrics+storage every 15 minutes and only persist and collected metrics from a timestamp if/when an error occurs.

Any idea what might be doing wrong with storage.local.retention?

[brian-brazil]

Is there an error or stack trace produced when it dies?

My aim is to clean the collected metrics+storage every 15 minutes and only persist and collected metrics from a timestamp if/when an error occurs.

It'd best to keep all the data for a good while. You never know when you need to retroactively debug.

[mokshpooja]

@brian-brazil not sure what this means.. is it looking for stored data to run the queries on?

prometheus_1 | time="2016-04-12T16:20:15Z" level=info msg="Maintenance loop stopped." file=storage.go line=858
prometheus_1 | panic: dropped unpersisted chunks from memory
prometheus_1 |
prometheus_1 | goroutine 117 [running]:
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*memorySeries).dropChunks(0xc209c429c0, 0x1540b450dbe)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:299 +0x31a
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).writeMemorySeries(0xc2080d6000, 0x8fb1b2e817d485e, 0xc209c429c0, 0x1540b450dbe, 0x7fb115ad8100)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:1043 +0x41d
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).maintainMemorySeries(0xc2080d6000, 0x8fb1b2e817d485e, 0x1540b450dbe, 0x0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:956 +0x284
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loop(0xc2080d6000)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:875 +0x310
prometheus_1 | created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:277 +0x51f
prometheus_1 |
prometheus_1 | goroutine 1 [select]:
prometheus_1 | main.Main(0x0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:153 +0x186c
prometheus_1 | main.main()
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:47 +0x1f
prometheus_1 |
prometheus_1 | goroutine 17 [syscall, 1 minutes]:
prometheus_1 | os/signal.loop()
prometheus_1 |  /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
prometheus_1 | created by os/signal.init·1
prometheus_1 |  /usr/local/go/src/os/signal/signal_unix.go:27 +0x35
prometheus_1 |
prometheus_1 | goroutine 89 [chan receive]:
prometheus_1 | main.func·005()
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:109 +0xbd
prometheus_1 | created by main.Main
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:112 +0xe4f
prometheus_1 |
prometheus_1 | goroutine 90 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc2080961c0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c
prometheus_1 |
prometheus_1 | goroutine 92 [select, 1 minutes]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b8160)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc
prometheus_1 |
prometheus_1 | goroutine 93 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b8160)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6
prometheus_1 |
prometheus_1 | goroutine 94 [select, 1 minutes]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b8160)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf
prometheus_1 |
prometheus_1 | goroutine 95 [select, 1 minutes]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b8160)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9
prometheus_1 |
prometheus_1 | goroutine 96 [select, 1 minutes]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b8160)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03
prometheus_1 |
prometheus_1 | goroutine 97 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc208096380)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c
prometheus_1 |
prometheus_1 | goroutine 98 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b82c0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc
prometheus_1 |
prometheus_1 | goroutine 99 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b82c0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6
prometheus_1 |
prometheus_1 | goroutine 100 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b82c0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf
prometheus_1 |
prometheus_1 | goroutine 101 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b82c0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9
prometheus_1 |
prometheus_1 | goroutine 102 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b82c0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03
prometheus_1 |
prometheus_1 | goroutine 103 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc208096700)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c
prometheus_1 |
prometheus_1 | goroutine 104 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b8420)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc
prometheus_1 |
prometheus_1 | goroutine 105 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b8420)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6
prometheus_1 |
prometheus_1 | goroutine 106 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b8420)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf
prometheus_1 |
prometheus_1 | goroutine 107 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b8420)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9
prometheus_1 |
prometheus_1 | goroutine 108 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b8420)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03
prometheus_1 |
prometheus_1 | goroutine 109 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc2080968c0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x21e
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x24c
prometheus_1 |
prometheus_1 | goroutine 110 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc2080b8580)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x55a
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7cc
prometheus_1 |
prometheus_1 | goroutine 111 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc2080b8580)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x14a
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7e6
prometheus_1 |
prometheus_1 | goroutine 112 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc2080b8580)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:768 +0x858
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9cf
prometheus_1 |
prometheus_1 | goroutine 113 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc2080b8580)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x283
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9e9
prometheus_1 |
prometheus_1 | goroutine 114 [select]:
prometheus_1 | github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc2080b8580)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x197
prometheus_1 | created by github.com/syndtr/goleveldb/leveldb.openDB
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa03
prometheus_1 |
prometheus_1 | goroutine 115 [select]:
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*persistence).processIndexingQueue(0xc2082b8800)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:1331 +0xba0
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*persistence).run(0xc2082b8800)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/persistence.go:282 +0x28
prometheus_1 | created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:253 +0x16a
prometheus_1 |
prometheus_1 | goroutine 116 [select]:
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).handleEvictList(0xc2080d6000)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:650 +0x46e
prometheus_1 | created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:276 +0x502
prometheus_1 |
prometheus_1 | goroutine 122 [select]:
prometheus_1 | github.com/prometheus/prometheus/rules.(*Manager).Run(0xc208011dc0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/rules/manager.go:159 +0x3c6
prometheus_1 | created by main.Main
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:135 +0x11d9
prometheus_1 |
prometheus_1 | goroutine 119 [select]:
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc2080d6000, 0x3d9, 0x3ff0000000000000, 0x1)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:754 +0x199
prometheus_1 | github.com/prometheus/prometheus/storage/local.func·043()
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:794 +0x2a1
prometheus_1 | created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughMemoryFingerprints
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:804 +0xfb
prometheus_1 |
prometheus_1 | goroutine 120 [select]:
prometheus_1 | github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).waitForNextFP(0xc2080d6000, 0x0, 0x3ff0000000000000, 0x0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:754 +0x199
prometheus_1 | github.com/prometheus/prometheus/storage/local.func·044()
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:827 +0x285
prometheus_1 | created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).cycleThroughArchivedFingerprints
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/storage.go:847 +0xfb
prometheus_1 |
prometheus_1 | goroutine 123 [chan receive]:
prometheus_1 | github.com/prometheus/prometheus/notification.(*NotificationHandler).Run(0xc208058280)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/notification/notification.go:187 +0x81
prometheus_1 | created by main.Main
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:138 +0x121c
prometheus_1 |
prometheus_1 | goroutine 640 [chan send]:
prometheus_1 | github.com/prometheus/prometheus/storage/local.func·030()
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:129 +0xec
prometheus_1 | created by github.com/prometheus/prometheus/storage/local.(*seriesMap).fpIter
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/local/series.go:134 +0xfb
prometheus_1 |
prometheus_1 | goroutine 125 [IO wait]:
prometheus_1 | net.(*pollDesc).Wait(0xc208447020, 0x72, 0x0, 0x0)
prometheus_1 |  /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
prometheus_1 | net.(*pollDesc).WaitRead(0xc208447020, 0x0, 0x0)
prometheus_1 |  /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
prometheus_1 | net.(*netFD).accept(0xc208446fc0, 0x0, 0x7fb115ad7e50, 0xc20841d438)
prometheus_1 |  /usr/local/go/src/net/fd_unix.go:419 +0x40b
prometheus_1 | net.(*TCPListener).AcceptTCP(0xc208050c48, 0x620b64, 0x0, 0x0)
prometheus_1 |  /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
prometheus_1 | net/http.tcpKeepAliveListener.Accept(0xc208050c48, 0x0, 0x0, 0x0, 0x0)
prometheus_1 |  /usr/local/go/src/net/http/server.go:1976 +0x4c
prometheus_1 | net/http.(*Server).Serve(0xc208069560, 0x7fb115adfa18, 0xc208050c48, 0x0, 0x0)
prometheus_1 |  /usr/local/go/src/net/http/server.go:1728 +0x92
prometheus_1 | net/http.(*Server).ListenAndServe(0xc208069560, 0x0, 0x0)
prometheus_1 |  /usr/local/go/src/net/http/server.go:1718 +0x154
prometheus_1 | net/http.ListenAndServe(0xb4ebe0, 0x5, 0x7fb115ade950, 0xc20801f580, 0x0, 0x0)
prometheus_1 |  /usr/local/go/src/net/http/server.go:1808 +0xba
prometheus_1 | github.com/prometheus/prometheus/web.(*Handler).Run(0xc20800a910)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/web/web.go:190 +0x151
prometheus_1 | created by main.Main
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:146 +0x12c5
prometheus_1 |
prometheus_1 | goroutine 128 [select]:
prometheus_1 | github.com/prometheus/prometheus/retrieval.(*Target).RunScraper(0xc20844c510, 0x7fb115ad8da8, 0xc20801f4a0)
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/target.go:289 +0x656
prometheus_1 | created by github.com/prometheus/prometheus/retrieval.(*TargetManager).updateTargetGroup
prometheus_1 |  /home/julius/gosrc/src/github.com/prometheus/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/retrieval/targetmanager.go:257 +0x96c
prometheus_1 |
prometheus_1 | goroutine 145 [syscall, locked to thread]:
prometheus_1 | runtime.goexit()
prometheus_1 |  /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1
vagrant_prometheus_1 exited with code 2

[juliusv]

@beorn7 See panic "panic: dropped unpersisted chunks from memory". Possible bug?

[brian-brazil]

That's a sanity check we have, in this case it's actually correct as the retention is too small. We can probably refine the check.

[juliusv]

Seems rather like a bug to me - if a user configures a very short retention period that overlaps with chunks being in memory, of course it should not crash, it should just keep the chunks as long as absolutely needed (like when they're still involved in an ongoing query), then drop them from everywhere. This kind of panic is usually only guarding against programming mistakes.

[beorn7]

Thanks for the trace. Yes, that looks like a bug. Will look into it ASAP.

[mokshpooja]

thanks guys, based on your discussion, it seems like this is caused when the retention is too small, so far I have been trying 1m-15m window. Just playing around right now to see how small a retention can be.

[beorn7]

@mokshpooja It's possible this is already fixed in head (or in 0.18rc1). I vaguely remember I have touched that part.
Needs to be investigated...

[mokshpooja]

@beorn7 what about the last stable release- 0.17.0? I have been using stable- 0.15.1.
Will check 0.18rc1

[juliusv]

Oh, please do try 0.18.0rc1 or HEAD. 0.15.1 is ancient and will have all kinds of bugs that are already fixed in master.

[beorn7]

Yes, definitely try to reproduce with 0.18rc1.

Which reminds me that we might want to write a Howto for filing bugs. It should include running from HEAD in master if possible, or running the latest published binaries (including RCs) as we tend to close issues once they are in master.

[mokshpooja]

@beorn7 I am trying 0.18rc1 from the binary tar file. Has anything changed in the prometheus.yml format? My old promtheus.yml is not working any more.

root@0052fef247a1:/prometheus# ./prometheus
prometheus, version 0.18.0rc1 (branch: release-0.18, revision: 0520414)
  build user:       root@b4961fdf73ed
  build date:       20160407-16:32:54
  go version:       go1.5.3
INFO[0000] Loading configuration file prometheus.yml     source=main.go:202
ERRO[0000] Couldn't load configuration (-config.file=prometheus.yml): unknown fields in global config: labels  source=main.go:214

My prometheus.yml-

global:
  scrape_interval: 15s 
  # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s 
  # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these extra labels to all time-series collected by this Prometheus instance.
  labels:
    monitor: 'project-Live monitoring'

rule_files:
  - '/alert.rules'

# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'cAdvisor_job'
    scrape_interval: 10s
    target_groups:
      - targets: ['cadvisor:8088']
        labels:
          group: 'cAdvisor'

[brian-brazil]

labels in global is now external_labels

[mokshpooja]

Hi Guys, now I have 0.18rc1 working however the problem remains the same. I have checked retention from 1-15 mins after the retention interval Prometheus still crashes. If you would like to see the massive error log let me know.
For now I will move my retention to a stable interval.

[beorn7]

Yes, please give us the crash log generated by 0.18rc1. If it's too long, as a gist.

[brian-brazil]

prometheus/storage/local/series.go

Line 323 in e83f05f

panic("dropped unpersisted chunks from memory")

looks to be it.

[mokshpooja]

Here it is- https://gist.github.com/mokshpooja/695e2ed4d9db133d5949adb82389d61f

[beorn7]

thanks. will look into it ASAP.

[beorn7]

I'll mark it as critical as with the new varbit chunks, a chunk may last so long that this could become in issue with longer retention times, too.

[mokshpooja]

thanks for the explanation, i think I understand what's going on here.

[beorn7]

OK, got it. So the problem is if the retention is shorter than the chunk timeout (currently hardcoded to 1h) or in the more extreme case shorter than the scrape interval. In both cases, the storage will try to drop a still open head chunk (because either it hasn't got a sample in a while but not long enough to close it because of the timeout, or because the last scrape is longer ago than the retention time quite regularly). An open head chunk cannot be persisted by definition, and that triggers our sanity check.

Good news is that this will strictly only happen with retention time below 1h. So it will not get worse with the longer lived varbit chunks. It should also be fairly simple to fix.

Stay tuned.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.