opening storage failed: block dir: : unexpected end of JSON input

[goldcode88]

Bug Report

What did you do?
start prometheus by command.
$/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.listen-address=xxx.xxx.xxx.xxx:yyyy --web.enable-admin-api

What did you expect to see?
I hope prometheus can print abnormal data directory, ignore them and boot normally.

What did you see instead? Under which circumstances?
It returns the following log.
level=info ts=2019-04-13T12:26:56.781358083Z caller=main.go:285 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-04-13T12:26:56.781562543Z caller=main.go:321 msg="Starting Prometheus" version="(version=2.8.0, branch=HEAD, revision=59369491cfdfe8dcb325723d6d28a837887a07b9)"
level=info ts=2019-04-13T12:26:56.781600089Z caller=main.go:322 build_context="(go=go1.11.5, user=root@4c4d5c29b71f, date=20190312-07:46:58)"
level=info ts=2019-04-13T12:26:56.781641101Z caller=main.go:323 host_details="(Linux 4.18.16-1.el7.elrepo.x86_64 #1 SMP Sat Oct 20 12:52:50 EDT 2018 x86_64 centos158 (none))"
level=info ts=2019-04-13T12:26:56.781678654Z caller=main.go:324 fd_limits="(soft=1024, hard=4096)"
level=info ts=2019-04-13T12:26:56.781712948Z caller=main.go:325 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-04-13T12:26:56.783144215Z caller=main.go:640 msg="Starting TSDB ..."
level=info ts=2019-04-13T12:26:56.783169598Z caller=web.go:418 component=web msg="Start listening for connections" address=xxx.xxx.xxx.xxx:yyyy
level=info ts=2019-04-13T12:26:56.784710972Z caller=main.go:509 msg="Stopping scrape discovery manager..."
level=info ts=2019-04-13T12:26:56.784733933Z caller=main.go:523 msg="Stopping notify discovery manager..."
level=info ts=2019-04-13T12:26:56.784743197Z caller=main.go:545 msg="Stopping scrape manager..."
level=info ts=2019-04-13T12:26:56.784755894Z caller=main.go:519 msg="Notify discovery manager stopped"
level=info ts=2019-04-13T12:26:56.784774309Z caller=main.go:505 msg="Scrape discovery manager stopped"
level=info ts=2019-04-13T12:26:56.784803397Z caller=manager.go:736 component="rule manager" msg="Stopping rule manager..."
level=info ts=2019-04-13T12:26:56.784848938Z caller=manager.go:742 component="rule manager" msg="Rule manager stopped"
level=info ts=2019-04-13T12:26:56.784873215Z caller=notifier.go:521 component=notifier msg="Stopping notification manager..."
level=info ts=2019-04-13T12:26:56.784899205Z caller=main.go:708 msg="Notifier manager stopped"
level=info ts=2019-04-13T12:26:56.785621752Z caller=main.go:539 msg="Scrape manager stopped"
level=error ts=2019-04-13T12:26:56.786112994Z caller=main.go:717 err="opening storage failed: block dir: "/var/lib/prometheus/01D6RR1N2G0J6YQ54EY5BNZE68": unexpected end of JSON input"

Environment

• System information:

uname -srm

Linux 4.18.16-1.el7.elrepo.x86_64 x86_64

• Prometheus version:
  prometheus, version 2.8.0 (branch: HEAD, revision: 5936949)
  build user: root@4c4d5c29b71f
  build date: 20190312-07:46:58
  go version: go1.11.5

After I remove all of "01D*" directories, prometheus boot normally.

[simonpasquier]

This looks very similar to #4058 which should be fixed in the latest Prometheus release (v2.9.0) by prometheus/tsdb#573.

I'm closing it for now, feel free to reopen it if you can reproduce with v2.9.0.