BatchSet runtime slice bounds out of range #308
Comments
|
What version of Badger are you using (presumably v0.8)? In your tests, you should create a new tmp dir, to isolate it from any previous runs, which might have issues; which isn't happening here. Based on the stack trace, the problem is with opening the KV, not BatchSet. Also, any reason not to use v1.0? |
|
@manishrjain thank you for the prompt reply, so am currently using the latest from master, at commit 7f945c8 and yes, that's probably v0.8. Sorry, I started using it a month ago so thought it was the latest. After every run, I am deleting the directory so that isolates previous failed runs. |
|
If you want to stick to the older version, can you please use tag v0.8.1, and retry? Do note that we're not actively supporting v0.8 series. |
|
@manishrjain same problem on v0.8.1 68f31e4 $ go test -v -bench=.
goos: darwin
goarch: amd64
BenchmarkIt-4 panic: runtime error: slice bounds out of range
goroutine 42 [running]:
github.com/dgraph-io/badger/table.(*blockIterator).parseKV(0xc431e9e1b0, 0xa86a00000, 0xffffffff)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:115 +0x4fa
github.com/dgraph-io/badger/table.(*blockIterator).Next(0xc431e9e1b0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:153 +0x1a3
github.com/dgraph-io/badger/table.(*blockIterator).Init(0xc431e9e1b0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:54 +0x3d
github.com/dgraph-io/badger/table.(*blockIterator).SeekToLast(0xc431e9e1b0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:105 +0x49
github.com/dgraph-io/badger/table.(*Iterator).seekToLast(0xc420011830)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:257 +0x139
github.com/dgraph-io/badger/table.(*Iterator).Rewind(0xc420011830)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:407 +0x33
github.com/dgraph-io/badger/table.OpenTable(0xc42ab56000, 0x1, 0x0, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/table.go:167 +0x220
github.com/dgraph-io/badger.(*KV).flushMemtable(0xc42010e300, 0xc4256980e0, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/kv.go:1121 +0x2cb
created by github.com/dgraph-io/badger.NewKV
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/kv.go:181 +0xcdb
exit status 2
FAIL _/Users/emmanuelodeke/Desktop/openSrc/bugs/badgerdb 0.324sAlright, I'll get the latest from and update the test code with it, Batches have disappeared and I think y'all replaced those with Txn's right? |
|
And yes I can still reproduce this even on the latest 6c5c932 with the exact same stack trace The repro modified is now package repro_test
import (
"crypto/rand"
"os"
"testing"
"github.com/dgraph-io/badger"
)
func randBytes(n int) []byte {
recv := make([]byte, n)
in, _ := rand.Reader.Read(recv)
return recv[:in]
}
var benchmarkData = []struct {
key, value []byte
}{
{randBytes(100), nil},
{randBytes(1000), []byte("foo")},
{[]byte("foo"), randBytes(1000)},
{[]byte(""), randBytes(1000)},
{nil, randBytes(1000000)},
{randBytes(100000), nil},
{randBytes(1000000), nil},
}
func BenchmarkIt(b *testing.B) {
dir := "testbadger_db"
os.RemoveAll(dir)
if err := os.MkdirAll(dir, 0755); err != nil {
b.Fatal(err)
}
defer os.RemoveAll(dir)
opts := new(badger.Options)
*opts = badger.DefaultOptions
opts.ValueLogFileSize = 1024 * 1024 * 1024
opts.Dir = dir
opts.ValueDir = dir
db, err := badger.Open(*opts)
if err != nil {
b.Fatal(err)
}
for i := 0; i < b.N; i++ {
tx := db.NewTransaction(true)
for _, kv := range benchmarkData {
tx.Set(kv.key, kv.value)
}
if err := tx.Commit(nil); err != nil {
b.Fatalf("#%d: batchSet err: %v", i, err)
}
}
}$ go test -v -bench=.
goos: darwin
goarch: amd64
BenchmarkIt-4 panic: runtime error: slice bounds out of range
goroutine 81 [running]:
github.com/dgraph-io/badger/table.(*blockIterator).parseKV(0xc4383c81b0, 0x386a80000, 0xffffffff)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:115 +0x4fa
github.com/dgraph-io/badger/table.(*blockIterator).Next(0xc4383c81b0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:153 +0x1a3
github.com/dgraph-io/badger/table.(*blockIterator).Init(0xc4383c81b0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:54 +0x3d
github.com/dgraph-io/badger/table.(*blockIterator).SeekToLast(0xc4383c81b0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:105 +0x49
github.com/dgraph-io/badger/table.(*Iterator).seekToLast(0xc420010480)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:257 +0x139
github.com/dgraph-io/badger/table.(*Iterator).Rewind(0xc420010480)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/iterator.go:407 +0x33
github.com/dgraph-io/badger/table.OpenTable(0xc4257d6018, 0x1, 0x0, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/table/table.go:166 +0x220
github.com/dgraph-io/badger.(*DB).flushMemtable(0xc42012a600, 0xc42000c360, 0x0, 0x0)
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/db.go:786 +0x2bf
created by github.com/dgraph-io/badger.Open
/Users/emmanuelodeke/go/src/github.com/dgraph-io/badger/db.go:253 +0xa58
exit status 2
FAIL _/Users/emmanuelodeke/Desktop/openSrc/bugs/badgerdb 0.691s |
Uncovered dgraph-io/badger#308 which perhaps will require us to use the latest BadgerDB since they no longer use v0.8(which we started with) plus until that issue is resolved, our benchmarks are invalid or spuriously error out.
Uncovered dgraph-io/badger#308 which perhaps will require us to use the latest BadgerDB since they no longer use v0.8(which we started with) plus until that issue is resolved, our benchmarks are invalid or spuriously error out.
|
Cut a release: https://github.com/dgraph-io/badger/releases/tag/v1.0.1. This should fix the issue you were seeing. Thanks for reporting it. Note that keys can't be longer than uint16, which this fix would now explicitly check. Update: You should check the result of |
|
Awesome, thank you @manishrjain for the quick fix and for the updates. A bit of 2 cents for the future, it is not necessary now but perhaps in the future: 1<<16 as a max key size might be limiting for folks that cache public keys and session-ids for example post-quantum cryptography which require higher security For now it is a far fetched example but the promises of your DB are very attractive that I can see myself and others using it in the near future for such purposes. |
|
Allowing There are ways around this limitation - one could fingerprint the session to generate key, and store the session itself as value. A case of collisions can be handled via transactions. |
|
True true, in deed once more folks ask for it then it'll become useful. Cheers man! |
|
No worries! Good luck with BadgerDB. |
Thank you very much for all the work on badgerDB, nice!
I'd like to report an issue that I noticed while doing work for @tendermint /cc @ebuchman @melekes:
Given the code below
If I run it, I consistently get a runtime panic
and the results of it seem weird after applying this patch, we get
Notice the logs
Ideally if
badger/table/iterator.go
Lines 112 to 114 in 6c5c932
executed, those logs would be different. Something weird is going on.
The text was updated successfully, but these errors were encountered: