Panic in newChunkForEncoding

[aecolley]

I had a prometheus which was crash-looping. On each restart it would complete crash recovery, wait for a minute, and then panic. The error message was always the same.

The host machine also suffered a disk-full condition about a day before, just like in #877 . Or, possibly, prometheus was SIGKILLed while it was still dealing with its SIGTERM. Either way, a write didn't happen, and recovery didn't recover.

Here's a clip of the stderr:

time="2015-11-27T08:12:38-05:00" level=warning msg="Crash recovery complete." source="crashrecovery.go:143" 
time="2015-11-27T08:12:38-05:00" level=info msg="18125 series loaded." source="storage.go:267" 
time="2015-11-27T08:12:38-05:00" level=info msg="Starting target manager..." source="targetmanager.go:114" 
time="2015-11-27T08:12:38-05:00" level=info msg="Listening on localhost:9090" source="web.go:223" 
time="2015-11-27T08:12:55-05:00" level=warning msg="Ignoring sample with out-of-order timestamp for fingerprint 9df8b97ea5a704cc (requests_total{<redacted>}): 1448629975.858 is not after 70369813640904.95" source="storage.go:570" 
panic: unknown chunk encoding: 212

goroutine 7425 [running]:
github.com/prometheus/prometheus/storage/local.newChunkForEncoding(0x7f4f5f1b72d4, 0x0, 0x0)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/storage/local/chunk.go:287 +0x259
github.com/prometheus/prometheus/storage/local.(*persistence).loadChunks(0xc8203e6300, 0x7a14e1da1e245688, 0xc828e92760, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/storage/local/persistence.go:435 +0x4e6
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).loadChunks(0xc8201c6000, 0x7a14e1da1e245688, 0xc828e92760, 0x2, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/storage/local/storage.go:1121 +0x8d
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunks(0xc824aad4a0, 0xc828e92730, 0x2, 0x2, 0x7a14e1da1e245688, 0xc8201c6000, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/storage/local/series.go:329 +0x3b7
github.com/prometheus/prometheus/storage/local.(*memorySeries).preloadChunksForRange(0xc824aad4a0, 0x1514913e33c, 0x1514913e33c, 0x7a14e1da1e245688, 0xc8201c6000, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/storage/local/series.go:419 +0x4fd
github.com/prometheus/prometheus/storage/local.(*memorySeriesStorage).preloadChunksForRange(0xc8201c6000, 0x7a14e1da1e245688, 0x1514913e33c, 0x1514913e33c, 0x45d964b800, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/storage/local/storage.go:646 +0x32c
github.com/prometheus/prometheus/storage/local.(*memorySeriesPreloader).PreloadRange(0xc82990c540, 0x7a14e1da1e245688, 0x1514913e33c, 0x1514913e33c, 0x45d964b800, 0x0, 0x0)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/storage/local/preload.go:34 +0x79
github.com/prometheus/prometheus/promql.(*Analyzer).Prepare(0xc826e7edc0, 0x7f4f5ebc1c00, 0xc828edc420, 0x0, 0x0, 0x0, 0x0)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/promql/analyzer.go:144 +0x628
github.com/prometheus/prometheus/promql.(*Engine).execEvalStmt(0xc82019b040, 0x7f4f5ebc1c00, 0xc828edc420, 0xc826ea8500, 0xc827bfbef0, 0x0, 0x0, 0x0, 0x0)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/promql/engine.go:386 +0x3be
github.com/prometheus/prometheus/promql.(*Engine).exec(0xc82019b040, 0xc826ea8500, 0x0, 0x0, 0x0, 0x0)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/promql/engine.go:357 +0x4ca
github.com/prometheus/prometheus/promql.(*query).Exec(0xc826ea8500, 0xc8290fe540)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/promql/engine.go:195 +0x2e
github.com/prometheus/prometheus/rules.(*AlertingRule).eval(0xc8206bb490, 0x1514913e33c, 0xc82019b040, 0x0, 0x0, 0x0, 0x0, 0x0)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/rules/alerting.go:157 +0x140
github.com/prometheus/prometheus/rules.(*Manager).runIteration.func1(0xc828ed7750, 0x1514913e33c, 0xc8201a8480, 0x7f4f5f133310, 0xc8206bb490)
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/rules/manager.go:252 +0xd3
created by github.com/prometheus/prometheus/rules.(*Manager).runIteration
    /Users/fabianreinartz/repos/src/github.com/prometheus/prometheus/rules/manager.go:278 +0x209

[fabxc]

@beorn7 seems like corruption caused by the disk running full. Can we just log and error and delete corrupted chunks as we encounter them?

[beorn7]

That's exactly what #877 is about.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.