The storage is now inconsistent. Restart Prometheus ASAP to initiate recovery

[adamdougal]

What did you do?
Shut down prometheus
What did you expect to see?
Clean shut down

What did you see instead? Under which circumstances?
This following error log: time="2017-08-04T13:12:17Z" level=error msg="The storage is now inconsistent. Restart Prometheus ASAP to initiate recovery." error="error in method hasArchivedMetric(cfe9908cdd5dcf2f): leveldb: closed" source="persistence.go:415"

Environment

• System information:

  Linux 4.4.0-89-generic x86_64

• Prometheus version:

   prometheus, version 1.7.1 (branch: master, revision: 		3afb3fffa3a29c3de865e1172fb740442e9d0133)
     build user:       root@0aa1b7fc430d
     build date:       20170612-11:44:05
     go version:       go1.8.3
  

• Logs:

   time="2017-08-04T13:09:08Z" level=warning msg="Received SIGTERM, exiting gracefully..." source="main.go:234"
   time="2017-08-04T13:09:08Z" level=info msg="Stopping local storage..." source="storage.go:457"
   time="2017-08-04T13:09:08Z" level=info msg="Stopping target manager..." source="targetmanager.go:77"
   time="2017-08-04T13:09:08Z" level=info msg="See you next time!" source="main.go:241"
   time="2017-08-04T13:09:08Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:633"
   time="2017-08-04T13:09:08Z" level=info msg="Stopping chunk eviction..." source="storage.go:467"
   time="2017-08-04T13:09:08Z" level=info msg="Stopping series quarantining..." source="storage.go:463"
   time="2017-08-04T13:09:08Z" level=info msg="Maintenance loop stopped." source="storage.go:1458"
   time="2017-08-04T13:09:08Z" level=info msg="Chunk eviction stopped." source="storage.go:1153"
   time="2017-08-04T13:09:08Z" level=info msg="Series quarantining stopped." source="storage.go:1907"
   time="2017-08-04T13:09:28Z" level=info msg="Checkpointing fingerprint mappings..." source="persistence.go:1526"
   time="2017-08-04T13:09:28Z" level=info msg="Done checkpointing in-memory metrics and chunks in 19.289224709s." source="persistence.go:665"
   time="2017-08-04T13:09:28Z" level=info msg="Done checkpointing fingerprint mappings in 363.915574ms." source="persistence.go:1549"
   time="2017-08-04T13:12:17Z" level=error msg="The storage is now inconsistent. Restart Prometheus ASAP to initiate recovery." error="error in method hasArchivedMetric(cfe9908cdd5dcf2f): leveldb: closed" source="persistence.go:415"
   time="2017-08-04T13:12:17Z" level=info msg="Local storage stopped." source="storage.go:484"
  

Looks very similar to #2509

[beorn7]

Is this happening repeatedly after you have gone through crash recovery once?

In general, there is a track record of subtle leveldb errors which are most likely due to corruptions on disk or bugs in leveldb. There is very little chance we will invest any work in investigating those as Prometheus 2 will not use LevelDB anymore. If the above issue goes away after going through crash recovery once, I'd take that as the appropriate work around for now.

[adamdougal]

Nope it's only happened the once so I'll close this and wait for Prometheus 2. Thanks!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.