Snapshot creation error: found chunk outside of compacted

[zuzzas]

Bug Report

What did you do?
Updated Prometheus v2.5.0 -> v2.6.0.
What did you expect to see?
A working snapshotting feature.
What did you see instead? Under which circumstances?
Snapshots stopped working, recovered backups from v2.5.0 aren't working as well.

{"status":"error","errorType":"internal","error":"create snapshot: snapshot head block: write compaction: found chunk with minTime: 1547726468168 maxTime: 1547726528168 outside of compacted minTime: 1547719200000 maxTime: 1547726527867"}

Environment

• System information:

  Linux 4.15.0-43-generic x86_64

• Prometheus version:

  prometheus, version 2.6.0 (branch: HEAD, revision: dbd1d58)
  build user: root@bf5760470f13
  build date: 20181217-15:14:46
  go version: go1.11.3

• Logs:

level=info ts=2019-01-17T12:00:00.045764719Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D0ZBFH0DZABWG7VYNNVTX9V7
level=info ts=2019-01-17T12:00:00.046032635Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D154WDRC1RJ4Z8BJPYYHCP4C
level=info ts=2019-01-17T12:00:00.046270873Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1AYA05AFGWN8AP8QCPBXP7Q
level=info ts=2019-01-17T12:00:00.046494941Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1CW28GMGNRRQDGN3DEWF0H9
level=info ts=2019-01-17T12:00:00.046644839Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DGMT7Q571MBSV66MG2M55C
level=info ts=2019-01-17T12:00:00.046755629Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DGKQWN3Q4K870EQTGCJ0CD
level=info ts=2019-01-17T12:00:00.046845693Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DQFF5Q5EXDTT5GRKB470SD
level=info ts=2019-01-17T12:00:27.62712416Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D0ZBFH0DZABWG7VYNNVTX9V7
level=info ts=2019-01-17T12:00:27.627376645Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D154WDRC1RJ4Z8BJPYYHCP4C
level=info ts=2019-01-17T12:00:27.627594562Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1AYA05AFGWN8AP8QCPBXP7Q
level=info ts=2019-01-17T12:00:27.627812374Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1CW28GMGNRRQDGN3DEWF0H9
level=info ts=2019-01-17T12:00:27.627949074Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DGMT7Q571MBSV66MG2M55C
level=info ts=2019-01-17T12:00:27.628038373Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DGKQWN3Q4K870EQTGCJ0CD
level=info ts=2019-01-17T12:00:27.628112874Z caller=db.go:787 component=tsdb msg="snapshotting block" block=01D1DQFF5Q5EXDTT5GRKB470SD

[simonpasquier]

What flags do you pass to Prometheus? Can you try using the tsdb CLI to check the data directory?

[zuzzas]

Will try tsdb tool.

/bin/prometheus --storage.tsdb.retention=1008h --storage.tsdb.path=/var/prometheus/data --config.file=/etc/prometheus/config/prometheus.yaml --web.route-prefix=/prometheus/ --web.external-url=https://test.flant.com/prometheus/ --web.console.libraries=/etc/prometheus/console_libraries --web.console.templates=/etc/prometheus/consoles --web.enable-lifecycle --web.enable-admin-api --rules.alert.resend-delay=30s

[gouthamve]

Hi @zuzzas, can you elaborate on "Snapshots stopped working"? I am able to snapshot and recover just fine on 2.6.1 which afaics has the same TSDB code as 2.6.

I'm looking into the restore from 2.5

[gouthamve]

Ah, I see the error now, sorry for missing it earlier.

[zuzzas]

@gouthamve
I've rm -rf'ed the entire Prometheus data directory. It still fails with the same error. I've tried turning on the debug log level, but there is nothing new. Could there be some strange metric that causes TSDB to misbehave?

[zuzzas]

Here's the interesting part. The snapshot works for all blocks, but the last one. Here is a graph from an attempt to restore such a snapshot to another Prometheus instance.

This snapshot was taken at 03:06.

[zuzzas]

I've resolved it by passing ?skip_head=true URL option to snapshot command. I lose a couple of hours in backup, but it works. Still, far from ideal.

[bwplotka]

Looks like prometheus/tsdb#514 should fix this.