Crash recovery should deal better with corrupt data for individual series in checkpoint file

[korovkin]

What did you do?

upgrade to newer version or prometheus

What did you expect to see?

clean start

What did you see instead? Under which circumstances?

crash

Environment

WARN[0007] Lost at least 9 chunks for fingerprint de012972ecd6f680, metric tunein_taps{country="DE", instance="localhost:9301", item="s236774", job="prometheus", source="browse"}.  source=crashrecovery.go:116
panic: runtime error: makeslice: cap out of range

goroutine 1 [running]:
panic(0x18b2120, 0xc446690260)
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/prometheus/prometheus/storage/local.(*persistence).recoverFromCrash(0xc420362c60, 0xc4200a90b0, 0x1, 0x17bdd20)
        /go/src/github.com/prometheus/prometheus/storage/local/crashrecovery.go:119 +0xb82
github.com/prometheus/prometheus/storage/local.(*persistence).loadSeriesMapAndHeads.func1(0xc420362c60, 0xc445e859e8, 0xc4200a90b0, 0xc445e859d8)
        /go/src/github.com/prometheus/prometheus/storage/local/persistence.go:817 +0xf6
github.com/prometheus/prometheus/storage/local.(*persistence).loadSeriesMapAndHeads(0xc420362c60, 0xc42000e500, 0x0, 0x267d540, 0xc420010190)
        /go/src/github.com/prometheus/prometheus/storage/local/persistence.go:840 +0x3f5
github.com/prometheus/prometheus/storage/local.(*MemorySeriesStorage).Start(0xc420238b00, 0x0, 0x0)
        /go/src/github.com/prometheus/prometheus/storage/local/storage.go:374 +0x1f5
main.Main(0x0)
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:181 +0x114f
main.main()
        /go/src/github.com/prometheus/prometheus/cmd/prometheus/main.go:43 +0x22

• System information:

uname -srm
Linux 4.4.0-62-generic x86_64

• Prometheus version:

./prometheus --version

prometheus, version 1.5.2 (branch: master, revision: bd1182d)
build user: root@a8af9200f95d
build date: 20170210-14:41:22
go version: go1.7.5

• Alertmanager version:

  insert output of alertmanager -version here (if relevant to the issue)

• Prometheus configuration file:

insert configuration here

[korovkin]

see
#1873 (comment)

for reference

[beorn7]

The code should handle this case more gracefully (by discarding this series and move on), but the root cause is a data corruption in the checkpoint file. Most likely you will have more corruptions, so even with the more graceful behavior in place, you might not be able to recover much.

[korovkin]

Got it. Should we close this issue? Or keep it opened ? sent from a 📞

…

On Mar 6, 2017, at 14:57, Björn Rabenstein ***@***.***> wrote: The code should handle this case more gracefully (by discarding this series and move on), but the root cause is a data corruption in the checkpoint file. Most likely you will have more corruptions, so even with the more graceful behavior in place, you might not be able to recover much. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[beorn7]

I'll take it as a reminder to fix the code as described above. Will change title.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.