Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

opening storage failed: block dir: : unexpected end of JSON input #5457

Closed
goldcode88 opened this issue Apr 13, 2019 · 3 comments

Comments

@goldcode88
Copy link

commented Apr 13, 2019

Bug Report

What did you do?
start prometheus by command.
$/usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus/ --web.listen-address=xxx.xxx.xxx.xxx:yyyy --web.enable-admin-api

What did you expect to see?
I hope prometheus can print abnormal data directory, ignore them and boot normally.

What did you see instead? Under which circumstances?
It returns the following log.
level=info ts=2019-04-13T12:26:56.781358083Z caller=main.go:285 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-04-13T12:26:56.781562543Z caller=main.go:321 msg="Starting Prometheus" version="(version=2.8.0, branch=HEAD, revision=59369491cfdfe8dcb325723d6d28a837887a07b9)"
level=info ts=2019-04-13T12:26:56.781600089Z caller=main.go:322 build_context="(go=go1.11.5, user=root@4c4d5c29b71f, date=20190312-07:46:58)"
level=info ts=2019-04-13T12:26:56.781641101Z caller=main.go:323 host_details="(Linux 4.18.16-1.el7.elrepo.x86_64 #1 SMP Sat Oct 20 12:52:50 EDT 2018 x86_64 centos158 (none))"
level=info ts=2019-04-13T12:26:56.781678654Z caller=main.go:324 fd_limits="(soft=1024, hard=4096)"
level=info ts=2019-04-13T12:26:56.781712948Z caller=main.go:325 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-04-13T12:26:56.783144215Z caller=main.go:640 msg="Starting TSDB ..."
level=info ts=2019-04-13T12:26:56.783169598Z caller=web.go:418 component=web msg="Start listening for connections" address=xxx.xxx.xxx.xxx:yyyy
level=info ts=2019-04-13T12:26:56.784710972Z caller=main.go:509 msg="Stopping scrape discovery manager..."
level=info ts=2019-04-13T12:26:56.784733933Z caller=main.go:523 msg="Stopping notify discovery manager..."
level=info ts=2019-04-13T12:26:56.784743197Z caller=main.go:545 msg="Stopping scrape manager..."
level=info ts=2019-04-13T12:26:56.784755894Z caller=main.go:519 msg="Notify discovery manager stopped"
level=info ts=2019-04-13T12:26:56.784774309Z caller=main.go:505 msg="Scrape discovery manager stopped"
level=info ts=2019-04-13T12:26:56.784803397Z caller=manager.go:736 component="rule manager" msg="Stopping rule manager..."
level=info ts=2019-04-13T12:26:56.784848938Z caller=manager.go:742 component="rule manager" msg="Rule manager stopped"
level=info ts=2019-04-13T12:26:56.784873215Z caller=notifier.go:521 component=notifier msg="Stopping notification manager..."
level=info ts=2019-04-13T12:26:56.784899205Z caller=main.go:708 msg="Notifier manager stopped"
level=info ts=2019-04-13T12:26:56.785621752Z caller=main.go:539 msg="Scrape manager stopped"
level=error ts=2019-04-13T12:26:56.786112994Z caller=main.go:717 err="opening storage failed: block dir: "/var/lib/prometheus/01D6RR1N2G0J6YQ54EY5BNZE68": unexpected end of JSON input"

Environment

  • System information:

uname -srm

Linux 4.18.16-1.el7.elrepo.x86_64 x86_64

  • Prometheus version:
    prometheus, version 2.8.0 (branch: HEAD, revision: 5936949)
    build user: root@4c4d5c29b71f
    build date: 20190312-07:46:58
    go version: go1.11.5

After I remove all of "01D*" directories, prometheus boot normally.

@simonpasquier

This comment has been minimized.

Copy link
Member

commented Apr 15, 2019

This looks very similar to #4058 which should be fixed in the latest Prometheus release (v2.9.0) by prometheus/tsdb#573.

I'm closing it for now, feel free to reopen it if you can reproduce with v2.9.0.

@stickyd

This comment has been minimized.

Copy link

commented May 7, 2019

@simonpasquier I'm having this issue on v2.9.0

I can manage a day or two without wiping the database, is there anything I can do to help out?

level=info ts=2019-05-07T17:49:12.504Z caller=main.go:285 msg="no time or size retention was set so using the default time retention" duration=15d
level=info ts=2019-05-07T17:49:12.504Z caller=main.go:321 msg="Starting Prometheus" version="(version=2.9.0, branch=HEAD, revision=1a9cdbd024fea05a1888c3164058087476cfdc3b)"
level=info ts=2019-05-07T17:49:12.504Z caller=main.go:322 build_context="(go=go1.12.3, user=root@4d7ed54ba6d9, date=20190415-09:06:49)"
level=info ts=2019-05-07T17:49:12.504Z caller=main.go:323 host_details="(Linux 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 prometheus-server-67dfcfdff4-mtbr9 (none))"
level=info ts=2019-05-07T17:49:12.504Z caller=main.go:324 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-05-07T17:49:12.504Z caller=main.go:325 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2019-05-07T17:49:12.505Z caller=main.go:640 msg="Starting TSDB ..."
level=info ts=2019-05-07T17:49:12.505Z caller=web.go:416 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1555848000000 maxt=1555912800000 ulid=01D924DQ5TXV3X8K0KK28Y07BW
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1555912800000 maxt=1555977600000 ulid=01D9427AZQAGFGTPYCASGBHQ6T
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1555977600000 maxt=1556042400000 ulid=01D9600SR88Z153205A014JHWH
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556042400000 maxt=1556107200000 ulid=01D97XTDPB9TZPMHX291K30DZR
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556107200000 maxt=1556172000000 ulid=01D99VKTFMVWCS3Z6Q3QRY2F6S
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556172000000 maxt=1556236800000 ulid=01D9BSDEWFWEKZ7WNY77W63N31
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556236800000 maxt=1556301600000 ulid=01D9DQ73DJB4MX55WR2HPJ30H7
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556301600000 maxt=1556366400000 ulid=01D9FN0PDP0ZCAA2K71M5W37PK
level=info ts=2019-05-07T17:49:12.505Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556366400000 maxt=1556431200000 ulid=01D9HJVCJDXJ9K8D19KVGM0J97
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556431200000 maxt=1556496000000 ulid=01D9KGM1KE8AYYTRKZD0BE8W6X
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556496000000 maxt=1556560800000 ulid=01D9NECTPJCSE08BKTAXN29X50
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556560800000 maxt=1556625600000 ulid=01D9QC66CMAVDQVSWRJQ03G1K2
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556625600000 maxt=1556690400000 ulid=01D9SA04QMW6KXD7VKZ5HB276W
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556690400000 maxt=1556755200000 ulid=01D9V7SVWSZC98W94CB1ZNKZ7Z
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556755200000 maxt=1556820000000 ulid=01D9X5KB71RHWYGX6XWQ9SAF08
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556820000000 maxt=1556884800000 ulid=01D9Z3CZV77H0WFR5M9K56TR9C
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556884800000 maxt=1556949600000 ulid=01DA116HZYKYJYYAG8KWYH0S0C
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1556949600000 maxt=1557014400000 ulid=01DA2Z0CR7D604G97YPR5J31JT
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1557014400000 maxt=1557079200000 ulid=01DA4WS37G2Y0JDFETR111YMAD
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1557144000000 maxt=1557151200000 ulid=01DA6TH8FCMYBE2B5HY7AH8KSM
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1557079200000 maxt=1557144000000 ulid=01DA6TJR8PR33W6A3F5YNAYBKN
level=info ts=2019-05-07T17:49:12.506Z caller=repair.go:47 component=tsdb msg="found healthy block" mint=1557151200000 maxt=1557158400000 ulid=01DA71CZPX4ZEX2VR1BJ2Z1CX2
level=info ts=2019-05-07T17:49:12.506Z caller=main.go:509 msg="Stopping scrape discovery manager..."
level=info ts=2019-05-07T17:49:12.506Z caller=main.go:523 msg="Stopping notify discovery manager..."
level=info ts=2019-05-07T17:49:12.506Z caller=main.go:545 msg="Stopping scrape manager..."
level=info ts=2019-05-07T17:49:12.506Z caller=main.go:519 msg="Notify discovery manager stopped"
level=info ts=2019-05-07T17:49:12.506Z caller=main.go:505 msg="Scrape discovery manager stopped"
level=info ts=2019-05-07T17:49:12.506Z caller=manager.go:743 component="rule manager" msg="Stopping rule manager..."
level=info ts=2019-05-07T17:49:12.506Z caller=manager.go:749 component="rule manager" msg="Rule manager stopped"
level=info ts=2019-05-07T17:49:12.506Z caller=notifier.go:522 component=notifier msg="Stopping notification manager..."
level=info ts=2019-05-07T17:49:12.506Z caller=main.go:708 msg="Notifier manager stopped"
level=info ts=2019-05-07T17:49:12.506Z caller=main.go:539 msg="Scrape manager stopped"
level=error ts=2019-05-07T17:49:12.506Z caller=main.go:717 err="opening storage failed: block dir: \"/data/01DA788Q0KTGBPCGFSWNJ4GYPQ\": unexpected end of JSON input"
@simonpasquier

This comment has been minimized.

Copy link
Member

commented May 9, 2019

@stickyd Can you move the discussion to the mailing list as using a closed issue isn't really convenient?
It would be useful to have the logs prior to the restart as well as the underlying filesystem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.