Loki: recover from panic opening boltdb files #2983
Conversation
…nics thrown by boltdb when opening a file.
| @@ -116,3 +119,13 @@ func CompressFile(src, dest string) error { | |||
|
|
|||
| return compressedFile.Sync() | |||
| } | |||
|
|
|||
| // SafeOpenBoltdbFile will recover from a panic opening a DB file, and return the panic message in the err return object. | |||
| func SafeOpenBoltdbFile(path string) (boltdb *bbolt.DB, err error) { | |||
There was a problem hiding this comment.
Should we add a flag to remove the underlying file? I'm worried that if we see a corrupted file we will no longer panic, but we'll continue to loop trying to open a broken file. In some cases, we can probably redownload it from storage instead.
edit: we could add this in a followup.
There was a problem hiding this comment.
i debated this as well, i even started adding some comments to add a little defense to this but even that wasn't straight forward.
As it is now all the code that calls this function had existing error handling for non-panic problems so I decided since we capture the panic and return it as an error it should also then get handled by any existing error handling logic which should already cover these cases.
Not totally in love with this myself, seems like we should try to centralize the behavior of what to do when having a problem with a boltdb file.
Improves #2970 but isn't really a fix, it's still not clear to me what causes this panic and why we have never seen it in any of the existing places we open DB files.