failure in newLevelsController ending in a fatal error

[vrischmann]

Hi,

I'm currently unable to use my badger db because of this fatal error.

Here's the log:

time="2019-01-10T21:15:25Z" level=fatal msg="Level validation: Levels Controller: Inter: \"\\xde\\x01g\\x9d\\xa50`\\x99F\\xe8\\\"k\\xc7tM\\x0f\\xd4\\xff\\xff\\xff\\xff\\xff\\xfb\\x8dn\" vs \"\\xde\\x01g\\x9d`n\\xd0\\xe4b\\x94z\\xd2\\nZflH\\xff\\xff\\xff\\xff\\xff\\xfcQ\\xfa\": level=2 j=18 numTables=24" package=db

I can reproduce it every time I open the database right now. Before it started the database closed correctly without crashing so I'm not sure what happened.

[manishrjain]

What version of Badger are you on? And, is there a way we can reproduce this issue at our end?

[manishrjain]

I could see this issue when I killed Badger. Not sure why it's happening. Haven't seen it ever before. Seems like a new bug.

[vrischmann]

I'm on your branch which speeds up Open.

Here's a repro program.
I ran it/killed it a couple of times until now where I have around 30Gib of data, I tried to relaunch it and it failed with this error:

badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000021.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000022.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000002.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000004.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000008.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000017.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000011.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000006.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000010.sst
badger 2019/01/11 16:02:48 INFO: Opening file: /home/vincent/tmp/badger-bug-data/000014.sst
badger 2019/01/11 16:02:48 INFO: All 2 tables opened in 51ms
FATA[0000] Level validation: Levels Controller: Inter:
00000000  39 39 39 39 39 ff ff ff  ff ff ff ff fe           |99999........|

 vs
00000000  21 62 61 64 67 65 72 21  68 65 61 64 ff ff ff ff  |!badger!head....|
00000010  ff fe d2 d7                                       |....|

: level=1 j=1 numTables=2

This is not private data so if need be I can share the database files but I don't know how I can share that amount of data.

I'll run the repro using master and see if I get the same error.

[vrischmann]

Ok I've been trying now for 15 minutes and I'm unable to have the crash using master.

[manishrjain]

I've been seeing it as well. Could be due to my branch. Trying to reproduce with the latest master.

[manishrjain]

Yeah. Can't reproduce it on master.