Prometheus v2.0.0-rc.0 "Opening storage failed" when used with k8s StatefulSets

[camilb]

Prometheus v2.0.0-rc.0 crashes when it is deployed in Kubernetes using StatefulSets.
This happens due the presence of lost+found directory in the mounted volume. After the lost+found directory is removed, the pods are started normally.

v2.0.0-beta.4 and v2.0.0-beta.5 doesn't have this issue.

Logs:

level=info ts=2017-10-06T10:27:57.505059945Z caller=main.go:213 msg="Starting prometheus" version="(version=2.0.0-rc.0, branch=HEAD, revision=012e52e3f9a0c1741b160498615a29bd5d723028)"
level=info ts=2017-10-06T10:27:57.505119915Z caller=main.go:214 build_context="(go=go1.9.1, user=root@d94ce23e8b9b, date=20171005-14:42:30)"
level=info ts=2017-10-06T10:27:57.505146886Z caller=main.go:215 host_details="(Linux 4.12.10-coreos #1 SMP Tue Sep 5 20:29:13 UTC 2017 x86_64 prometheus-k8s-0 (none))"
level=info ts=2017-10-06T10:27:57.509750537Z caller=web.go:378 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2017-10-06T10:27:57.510386074Z caller=main.go:307 msg="Starting TSDB"
level=info ts=2017-10-06T10:27:57.511306322Z caller=targetmanager.go:68 component="target manager" msg="Starting target manager..."
level=error ts=2017-10-06T10:27:57.522333344Z caller=main.go:316 msg="Opening storage failed" err="open block /var/prometheus/data/01BVPQNJ945N7GQTDPKFP53YMA: read label index table: invalid size"

lost+found

[grobie]

Have you purged the storage after migrating from beta.5 to rc.0? There were incompatible storage changes.

[camilb]

@grobie This happens with a fresh deployment.

[camilb]

@grobie It looks that Kubernetes kept the volumes after deleting the StatefulSets. Manually removed the persistent volumes and deployed again. Works fine, thanks!

[cauwulixuan]

I met the same issue. I removed PV manually and create PV, PVC again, still met this issue. Any suggestions?

[mclarke47]

@WuLeeX You can use the volume mount permissions hack until this is fixed EXAMPLE.

[omerfarukz]

Try yo delete or rename lock file in data directory.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.