Slice bounds out of range error

[pawanrawal]

Steps to reproduce

1. Compile and run dgraph on branch badger.
   ./dgraph

2. Load goldendata using dgraphloader.
   ./dgraphloader -r ~/go/src/github.com/dgraph-io/benchmarks/data/goldendata.rdf.gz

3. As soon as dgraphloader finishes, issue a shutdown command to server.
   curl localhost:8080/admin/shutdown

4. Start Dgraph again.
   ./dgraph

5. Now try taking a backup.
   curl localhost:8080/admin/backup

Dgraph panics with error

panic: runtime error: slice bounds out of range

goroutine 19899 [running]:
github.com/dgraph-io/dgraph/x.Parse(0xc43331498a, 0x6, 0x6, 0x2)
	/home/pawan/go/src/github.com/dgraph-io/dgraph/x/keys.go:195 +0x236
github.com/dgraph-io/dgraph/worker.backup(0x1, 0xcd16da, 0x6, 0x0, 0x0)
	/home/pawan/go/src/github.com/dgraph-io/dgraph/worker/backup.go:254 +0x85e
github.com/dgraph-io/dgraph/worker.handleBackupForGroup(0x7f6d936a4970, 0xc4200105b0, 0x627d3dd58c69d20, 0xc400000001, 0x0)
	/home/pawan/go/src/github.com/dgraph-io/dgraph/worker/backup.go:330 +0xa83
github.com/dgraph-io/dgraph/worker.BackupOverNetwork.func1(0xc42b4f6d20, 0x7f6d936a4970, 0xc4200105b0, 0x1)
	/home/pawan/go/src/github.com/dgraph-io/dgraph/worker/backup.go:445 +0x4b
created by github.com/dgraph-io/dgraph/worker.BackupOverNetwork
	/home/pawan/go/src/github.com/dgraph-io/dgraph/worker/backup.go:446 +0x138

[szymonm]

Any idea how is this badger related, @pawanrawal?

[pawanrawal]

I am sure that it is related to badger because I couldn't reproduce it on master. I suspect somehow incomplete data for the key is getting written on disk. When we try to read it back in Dgraph and parse it we get out of bounds error. I have set syncWrites to true.

There could also be something wrong with how its used within Dgraph https://github.com/dgraph-io/dgraph/blob/badger/worker/backup.go#L258.

[pawanrawal]

This is caused by the /head/ key. While taking backup we go through all keys and try to parse them. Since this key wasn't written from Dgraph we cant parse it and hence get this error. This should ideally not be returned to the user during iteration.

[manishrjain]

Agreed. Can you change the badger iterator to handle this case? Sent from Nexus 6P

…

On May 20, 2017 6:54 PM, "Pawan Rawal" ***@***.***> wrote: This is caused by the /head/ key. While taking backup we go through all keys and try to parse them. Since this key wasn't written from Dgraph we cant parse it and hence get this error. This should ideally not be returned to the user during iteration. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#37 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABsyNNH0LXyDw3ZPro7NpYsw0mPcC3g0ks5r7qpegaJpZM4NgZSV> .

[pawanrawal]

Sure, should we write the /head/ key to some other file separate from the .sst files? That way we can avoid this special check in iterator functions.

[manishrjain]

No, it has to go along with the .sst files. That's the only way to know how far LSM tree has reached.

Update: You can also do some optimization, so if we are already past the /head/ key, we just store that and don't keep on comparing the keys.

[manishrjain]

This should now be fixed.