Index out of range crash

[cespare]

Hey, I've run into a crashing issue. It may be my fault, but my usage is pretty basic. I discovered this after doing some tests where I inserted many thousands of chat messages into Bolt, but fortunately I was able to reproduce with randomly generated messages as well.

Here's the repro code:

https://gist.github.com/cespare/9623b31999bbc0e5266a

All it does is insert 1000 20-word randomly generated messages into a Bolt database, keyed by an incrementing ID.

When you run this once, it's fine. On the second run, doing anything after opening the DB causes a crash. In my repro code I used db.Buckets() to cause the crash.

Note that if you insert fewer documents (say, 900), then the crash doesn't happen.

[cespare]

Here's a slightly nicer repro case:

https://gist.github.com/cespare/caef571fd9c8d3db72c2

Here I closed and reopened the DB from within my code so you only need to run it once.

• Open the DB
• Write 1000 documents
• Close and reopen
• Call db.Buckets() crashes

[tv42]

It looks like when opening a file, it's mmapped at whatever size the file naturally has, and then the Buckets call tries to access a page beyond that point, getting an index out of bounds panic.

Not 100% on what's the best way to fix this. It seems the db size / page count is not stored explicitly in the meta page? Otherwise, could read that to get the right mmap size, when opening. Alternatively, always write to last byte when growing -- but even then, it's not guaranteed to hit the disk immediately.

[benbjohnson]

@cespare @tv42 It was a really dumb bug. While calculating the minimum size I used a wrong intermediate variable. It's fixed with #55.

[benbjohnson]

@cespare by the way, thanks for the reproducible code. It made it really easy to fix the bug.

[cespare]

@benbjohnson Thanks!

[benbjohnson]

@tv42 The meta page doesn't store the last byte written but it does store a high water mark for pages using Meta.pgid. You can calculate the last byte by multiplying that by the Meta.pageSize.

[tv42]

I see what's going on with meta.pgid, I confused that with page.id when reading so I was blind to it. I just went through the meta0 vs meta1 and transaction handling code -- really sweet & simple!

[benbjohnson]

@tv42 I really like the simple approach LMDB used with the "double buffering" meta pages. It's hard to get much simpler than that. :)

I should probably document some of the unexported fields (e.g. meta.pgid, etc). Or maybe better names would help like meta.maxpgid. Actually, I need to fix some of the naming. I tried to Go-ify the names as I ported it over but I didn't touch the lower level pieces.

Maybe the following changes in Meta to start:

type Meta struct {
    ...
    bucketPageID  pageID
    freelistPageID pageID
    maxPageID     pageID
    maxTransactionID    transactionID
}