Badger shouldn't just panic when open a bad database #1797

HaoboGu · 2022-07-28T06:45:03Z

When BadgerDB tries to open a corrupted dababase, sometime it would just panic(seems that this panic only occurs on Windows):

panic: runtime error: index out of range [3] with length 0 [recovered]
	panic: 
== Recovering from initIndex crash ==
File Info: [ID: 38, Size: 408, Zeros: 408]
isEnrypted: false checksumLen: 0 checksum:  indexLen: 0 index: [] 
== Recovered ==


goroutine 80 [running]:
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1.1(0xc0001c1aa0)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:351 +0x165
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest.func1(0xc0001d6180)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:397 +0x6e
panic(0x1d77ca0, 0xc0003bcd68)
	C:/go/src/runtime/panic.go:971 +0x49a
github.com/google/flatbuffers/go.GetInt32(...)
	C:/gocode/pkg/mod/github.com/google/flatbuffers@v1.12.0/go/encode.go:85
github.com/google/flatbuffers/go.GetUOffsetT(...)
	C:/gocode/pkg/mod/github.com/google/flatbuffers@v1.12.0/go/encode.go:121
github.com/dgraph-io/badger/v3/fb.GetRootAsTableIndex(...)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/fb/TableIndex.go:14
github.com/dgraph-io/badger/v3/table.(*Table).readTableIndex(0xc0001d6180, 0x0, 0x8, 0xc0001c1a40)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:704 +0x491
github.com/dgraph-io/badger/v3/table.(*Table).initIndex(0xc0001d6180, 0x0, 0x0, 0xc0002a3260)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:462 +0x545
github.com/dgraph-io/badger/v3/table.(*Table).initBiggestAndSmallest(0xc0001d6180, 0x0, 0x0)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:401 +0xab
github.com/dgraph-io/badger/v3/table.OpenTable(0xc000434280, 0x0, 0x200000, 0x0, 0x0, 0x3f847ae147ae147b, 0x1000, 0x0, 0x1, 0xc00053a080, ...)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/table/table.go:308 +0x36e
github.com/dgraph-io/badger/v3.newLevelsController.func1(0xc00043e180, 0xc0002b57b0, 0xc000589000, 0xc0002b5798, 0xc000178d10, 0x7, 0x7, 0xc00050a780, 0x21, 0x6, ...)
	C:/gocode/pkg/mod/github.com/dgraph-io/badger/v3@v3.2011.1/levels.go:150 +0x2d8
created by github.com/dgraph-io/badger/v3.newLevelsController

As a widely used dependency, BadgerDB should not just panic. Those panics cannot be captured and then breaks the whole application.

The text was updated successfully, but these errors were encountered:

github-actions · 2022-07-28T06:45:16Z

Use Discuss Issues for reporting issues about this repository.

HaoboGu · 2022-07-28T06:46:36Z

@skrdgraph it's the issue, much appreciated if you can help

skrdgraph · 2022-07-28T06:48:28Z

@HaoboGu will do our best. For the time being, I'm tagging @pandeyshubham25 (he may have come across this) ...

HaoboGu · 2022-07-28T06:55:51Z

thanks very much!

skrdgraph · 2022-07-28T06:57:22Z

@HaoboGu would you be able to provide us with sample steps to reproduce this issue?

some other qs:

details about your badger version # or tag or commit-sha?
are you seeing this issue with latest release of badger?
system specs would be helpful
detailed steps to reproduce the issue, it would be good to get sample data for this issue (possibly)
detail expected behavior (you have described it partially in the issue, a little more Information may help)

HaoboGu · 2022-07-28T07:04:19Z

badger version: github.com/dgraph-io/badger/v3 v3.2103.2
Yes, I'm using latest release version of badger
Windows 10 1903 (OS build 18362.1082) / go1.16.2 windows/amd64
reproduce:

download the corrupted db
db.zip
open it on Windows using badger.Open(badger.DefaultOptions(dbPath).WithVerifyValueChecksum(true))
badger panics then

badger should return an error instead of just panic

skrdgraph · 2022-07-28T08:26:55Z

i'll try this later in the day, and circle back. pls bear with me as I figure this out (will be discussing with the internal team on this ...)

HaoboGu · 2022-07-28T08:29:52Z

no problem :D

skrdgraph · 2022-07-28T09:29:56Z

ok this may not be very helpful, tested this on Mac-OS 12.4 (won't have access to windows machine till tomorrow)

Looks like I get a different error while trying to do this:

$ badger  --dir ~/Downloads/db/ info show-keys
Listening for /debug HTTP requests at port: 8080
jemalloc enabled: false
Using Go memory

[     2021-05-05T15:08:08Z] MANIFEST      716 B MA
[          1 month earlier] 000002.sst    247 B L6 
[          1 month earlier] 000004.sst    246 B L6 
[          1 month earlier] 000006.sst    247 B L6 
[          1 month earlier] 000008.sst    247 B L6 
[          1 month earlier] 000010.sst    247 B L6 
[          1 month earlier] 000012.sst    247 B L6 
[          1 month earlier] 000014.sst    247 B L6 
[          1 month earlier] 000016.sst    247 B L6 
[          1 month earlier] 000018.sst    247 B L6 
[          1 month earlier] 000023.sst    337 B L6 
[          3 weeks earlier] 000025.sst    247 B L6 
[         21 hours earlier] 000028.sst    408 B L6 
[         21 hours earlier] 000030.sst    247 B L6 
[         21 hours earlier] 000032.sst    247 B L6 
[         19 hours earlier] 000035.sst    410 B L6 
[       30 minutes earlier] 000038.sst    408 B L6 
[       29 minutes earlier] 000041.vlog    20 B VL
[       29 minutes earlier] 000042.vlog  2.0 GiB VL

[EXTRA]
[2021-05-05T15:08:08Z] 00086.mem    128 MiB
[2021-03-08T17:42:34Z] DISCARD      1.0 MiB
[2021-03-08T17:42:34Z] KEYREGISTRY    28 B

[Summary]
Level 0 size:          0 B
Level 1 size:          0 B
Level 2 size:          0 B
Level 3 size:          0 B
Level 4 size:          0 B
Level 5 size:          0 B
Level 6 size:      4.4 KiB
Total SST size:    4.4 KiB
Value log size:    2.0 GiB

Abnormalities:
3 extra files.
0 missing files.
0 empty files.
0 truncated manifests.
Error: failed to open database err: while opening memtables error: while opening fid: 86 error: while updating skiplist error: end offset: 20 < size: 134217728 error: Log truncate required to run DB. This might result in data loss
github.com/dgraph-io/badger/v3.init
	/Users/sudhishkr/scratch/github/dgraph-io/badger/errors.go:101
runtime.doInit
	/usr/local/opt/go/libexec/src/runtime/proc.go:6222
runtime.doInit
	/usr/local/opt/go/libexec/src/runtime/proc.go:6199
runtime.doInit
	/usr/local/opt/go/libexec/src/runtime/proc.go:6199
runtime.main
	/usr/local/opt/go/libexec/src/runtime/proc.go:233
runtime.goexit
	/usr/local/opt/go/libexec/src/runtime/asm_amd64.s:1571

Are you trying to open the DB like this? badger --dir ~/Downloads/db/ info show-keys <<

HaoboGu · 2022-07-28T09:44:59Z

I got this:

PS C:\Users\ADMIN\Downloads> badger --dir db info show-keys
Listening for /debug HTTP requests at port: 8080
jemalloc enabled: false
Using Go memory

[2021-05-05T15:08:07+08:00] MANIFEST      716 B MA
[          1 month earlier] 000002.sst    247 B L6 
[          1 month earlier] 000004.sst    246 B L6 
[          1 month earlier] 000006.sst    247 B L6 
[          1 month earlier] 000008.sst    247 B L6 
[          1 month earlier] 000010.sst    247 B L6 
[          1 month earlier] 000012.sst    247 B L6 
[          1 month earlier] 000014.sst    247 B L6 
[          1 month earlier] 000016.sst    247 B L6 
[          1 month earlier] 000018.sst    247 B L6 
[          1 month earlier] 000023.sst    337 B L6 
[          3 weeks earlier] 000025.sst    247 B L6 
[         21 hours earlier] 000028.sst    408 B L6 
[         21 hours earlier] 000030.sst    247 B L6 
[         21 hours earlier] 000032.sst    247 B L6 
[         19 hours earlier] 000035.sst    410 B L6 
[       30 minutes earlier] 000038.sst    408 B L6 
[       29 minutes earlier] 000041.vlog    20 B VL
[       29 minutes earlier] 000042.vlog  2.0 GiB VL

[EXTRA]
[2021-05-05T15:08:07+08:00] 00086.mem    128 MiB
[2021-03-08T17:42:33+08:00] DISCARD      1.0 MiB
[2021-03-08T17:42:32+08:00] KEYREGISTRY    28 B

[Summary]
Level 0 size:          0 B
Level 1 size:          0 B
Level 2 size:          0 B
Level 3 size:          0 B
Level 4 size:          0 B
Level 5 size:          0 B
Level 6 size:      4.4 KiB
Total SST size:    4.4 KiB
Value log size:    2.0 GiB

Abnormalities:
3 extra files.
0 missing files.
0 empty files.
0 truncated manifests.
Error: failed to open database err: Read-only mode is not supported on Windows
github.com/dgraph-io/badger/v3.init
        C:/Users/ADMIN/Downloads/badger-3.2103.2/errors.go:94
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6222
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6199
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6199
runtime.main
        C:/Program Files/Go/src/runtime/proc.go:233
runtime.goexit
        C:/Program Files/Go/src/runtime/asm_amd64.s:1571
Usage:
  badger info [flags]

Flags:
      --cv-mode string       [none, table, block, tableAndBlock] Specifies when the db should verify checksum for SST. (default "none")
      --discard              Parse and print DISCARD file from value logs.
      --enc-key string       Use the provided encryption key
  -h, --help                 help for info
      --histogram            Show a histogram of the key and value sizes.
      --history              Show all versions of a key
  -l, --lookup string        Hex of the key to lookup
      --read-only            If set to true, DB will be opened in read only mode. If DB has not been closed properly, this option can be set to false to open DB. (default true)
      --show-internal        Show internal keys along with other keys. This option should be used along with --show-key option
      --show-keys            Show keys stored in Badger
      --show-meta            Output item meta data as well (default true)
  -s, --show-tables          If set to true, show tables as well.       
      --truncate             If set to true, it allows truncation of value log files if they have corrupt data.
      --with-prefix string   Consider only the keys with specified prefix

Global Flags:
      --dir string        Directory where the LSM tree files are located. (required)
      --vlog-dir string   Directory where the value log files are located, if different from --dir

failed to open database err: Read-only mode is not supported on Windows
github.com/dgraph-io/badger/v3.init
        C:/Users/ADMIN/Downloads/badger-3.2103.2/errors.go:94
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6222
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6199
runtime.doInit
        C:/Program Files/Go/src/runtime/proc.go:6199
runtime.main
        C:/Program Files/Go/src/runtime/proc.go:233
runtime.goexit
        C:/Program Files/Go/src/runtime/asm_amd64.s:1571

skrdgraph · 2022-07-28T09:46:34Z

your error is probably related to folder permissions -> failed to open database err: Read-only mode is not supported on Windows.

my error is something I need to look further into. @MichelDiz have you see this before?

@HaoboGu - how did you exactly simulate the error you have indicated in your original post #1797 (comment)? You did respond saying

open it on Windows using badger.Open(badger.DefaultOptions(dbPath).WithVerifyValueChecksum(true))

if you could elaborate on the steps it will be helpful.

HaoboGu · 2022-07-28T09:56:11Z

This is the minimal repro that I can do:

Here is the main.go:

package main

import (
	"fmt"

	"github.com/dgraph-io/badger/v3"
)

func main() {
	opts := badger.DefaultOptions("C://Users/ADMIN/Downloads/db")
	_, err := badger.Open(opts)
	if err != nil {
		fmt.Print(err)
	}
}

Then go run ./main.go

BenChen-PKI · 2022-12-09T05:41:56Z

I would suggest in function newLevelsController of levels.go, add a defer function to recover and panic again in the go routine (line 129), so that the panic in this go routine can be caught by the client code.
in this way, the application can handle the panic from badgerDB when opening a corrupted db, then delete the corrupted db and create a new db to serve.

github-actions bot closed this as completed Jul 28, 2022

joshua-goldstein reopened this Aug 4, 2022

mangalaman93 mentioned this issue Mar 18, 2023

[BUG]: Panic after power fail #1883

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Badger shouldn't just panic when open a bad database #1797

Badger shouldn't just panic when open a bad database #1797

HaoboGu commented Jul 28, 2022

github-actions bot commented Jul 28, 2022

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022 •

edited

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022 •

edited

HaoboGu commented Jul 28, 2022 •

edited

BenChen-PKI commented Dec 9, 2022

Badger shouldn't just panic when open a bad database #1797

Badger shouldn't just panic when open a bad database #1797

Comments

HaoboGu commented Jul 28, 2022

github-actions bot commented Jul 28, 2022

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022 • edited

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022

HaoboGu commented Jul 28, 2022

skrdgraph commented Jul 28, 2022 • edited

HaoboGu commented Jul 28, 2022 • edited

BenChen-PKI commented Dec 9, 2022

skrdgraph commented Jul 28, 2022 •

edited

skrdgraph commented Jul 28, 2022 •

edited

HaoboGu commented Jul 28, 2022 •

edited