Prometheus panicing with panic: runtime error: index out of range

[brian-brazil]

My prometheus is in a crashloop, this is following a successful recovery:

prometheus, version 0.9.0rc2 (branch: HEAD, revision: 917acb6)
  build user:       boxever@boxever-thor
  build date:       20150106-11:23:53
  go version:       1.4
panic: runtime error: index out of range

goroutine 212 [running]:
github.com/prometheus/prometheus/storage/local.(*memorySeries).lastTime(0xc21d164e00, 0x14a3eba9ded)
        /home/boxever/.jenkins/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/loca
l/series.go:417 +0x6a
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).maintainSeries(0xc227eac540, 0x5234c148c4be82e4)
        /home/boxever/.jenkins/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/loca
l/storage.go:700 +0x21d
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loop(0xc227eac540)
        /home/boxever/.jenkins/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/loca
l/storage.go:656 +0x330
created by github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).Start
        /home/boxever/.jenkins/workspace/prometheus/.build/root/gopath/src/github.com/prometheus/prometheus/storage/loca
l/storage.go:190 +0x66

Log file created at: 2015/01/06 10:38:53
Running on machine: ip-10-51-39-88
Binary: Built with gc go1.4 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0106 10:38:53.164537   23955 targetmanager.go:62] Pool for job prometheus does not exist; creating and starting...
I0106 10:38:53.165238   23955 targetmanager.go:62] Pool for job alertmanager does not exist; creating and starting...
I0106 10:38:53.165253   23955 targetmanager.go:62] Pool for job pushgateway does not exist; creating and starting...
I0106 10:38:53.165304   23955 targetmanager.go:62] Pool for job node does not exist; creating and starting...
I0106 10:38:53.166318   23955 targetmanager.go:62] Pool for job haproxy does not exist; creating and starting...
I0106 10:38:53.166350   23955 targetmanager.go:62] Pool for job zookeeper does not exist; creating and starting...
I0106 10:38:53.166367   23955 targetmanager.go:62] Pool for job kafka does not exist; creating and starting...
I0106 10:38:53.166384   23955 targetmanager.go:62] Pool for job boxever_is does not exist; creating and starting...
I0106 10:38:53.166411   23955 targetmanager.go:62] Pool for job boxever_ras does not exist; creating and starting...
I0106 10:38:53.166458   23955 targetmanager.go:62] Pool for job boxever_de does not exist; creating and starting...
I0106 10:38:53.166473   23955 targetmanager.go:62] Pool for job boxever_rt does not exist; creating and starting...
I0106 10:38:53.166502   23955 targetmanager.go:62] Pool for job boxever_demo does not exist; creating and starting...
I0106 10:38:53.166518   23955 targetmanager.go:62] Pool for job boxever_st does not exist; creating and starting...
I0106 10:38:53.166641   23955 targetmanager.go:62] Pool for job boxever_bl does not exist; creating and starting...
I0106 10:38:53.166664   23955 targetmanager.go:62] Pool for job boxever_ae does not exist; creating and starting...
I0106 10:38:53.166681   23955 targetmanager.go:62] Pool for job boxever_guests_service does not exist; creating and star
ting...
I0106 10:38:53.196600   23955 storage.go:107] Loading series map and head chunks...
W0106 10:38:56.494126   23955 persistence.go:964] Persistence layer appears dirty.
W0106 10:38:56.494547   23955 persistence.go:267] Starting crash recovery. Prometheus is inoperational until complete.
I0106 10:38:56.494557   23955 persistence.go:273] Scanning files.
W0106 10:38:56.496351   23955 persistence.go:438] Treating recovered metric instance_device:node_disk_reads_completed:ra
te5m{device="xvdb", instance="http://ip-10-82-144-167:3002/metrics", job="node"}, fingerprint 005fae92886c9be7, as fresh
ly unarchived, with 111 chunks in series file.
.
.
.
W0106 10:38:59.213742   23955 persistence.go:438] Treating recovered metric zk_approximate_data_size{instance="http://ip-10-80-89-141:2180/metrics", job="zookeeper"}, fingerprint ffbda3523759a097, as freshly unarchived, with 134 chunks in series file.
I0106 10:38:59.213910   23955 persistence.go:300] File scan complete. 44972 series found.
I0106 10:38:59.213919   23955 persistence.go:302] Checking for series without series file.
I0106 10:38:59.220632   23955 persistence.go:339] Check for series without series file complete.
I0106 10:38:59.220656   23955 persistence.go:505] Cleaning up archive indexes.
I0106 10:38:59.456161   23955 persistence.go:512] 10000 archived metrics checked.
I0106 10:38:59.876451   23955 persistence.go:571] 10000 archived time ranges checked.
I0106 10:39:00.026128   23955 persistence.go:591] Clean-up of archive indexes complete.
I0106 10:39:00.026149   23955 persistence.go:599] Rebuilding label indexes.
I0106 10:39:00.026154   23955 persistence.go:600] Indexing metrics in memory.
I0106 10:39:00.031183   23955 persistence.go:605] 10000 metrics queued for indexing.
I0106 10:39:00.060866   23955 persistence.go:605] 20000 metrics queued for indexing.
I0106 10:39:00.168951   23955 persistence.go:608] Indexing archived metrics.
I0106 10:39:00.198804   23955 persistence.go:621] 30000 metrics queued for indexing.
I0106 10:39:00.317057   23955 persistence.go:621] 40000 metrics queued for indexing.
I0106 10:39:00.362127   23955 persistence.go:627] All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)
W0106 10:39:00.362149   23955 persistence.go:349] Crash recovery complete.
I0106 10:39:00.366965   23955 storage.go:112] 26706 series loaded.
W0106 10:39:00.368922   23955 main.go:142] No TSDB URL provided; not sending any samples to long-term storage
I0106 10:39:00.388559   23955 web.go:97] listening on :3000
I0106 10:39:10.386149   23955 storage.go:629] Completed maintenance sweep through archived fingerprints in 1.185µs.
I0106 10:39:20.399985   23955 storage.go:629] Completed maintenance sweep through archived fingerprints in 1.334µs.

[brian-brazil]

Stracing doesn't point to any obvious culprit.

[juliusv]

Thanks for reporting! Closing, as this is a dupe of #435. We'll address this before an announcement, of course.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.