Crash when online write

[JimWen]

What version of Dgraph are you using?

Dgraph version : v1.2.4
Dgraph SHA-256 : 38ca7ecc19103d21bdd1eddf0d2695e872883b37d4c270916606b0b51f1d0a1b
Commit SHA-1 : b51171c
Commit timestamp : 2020-05-15 15:42:10 -0700
Branch : HEAD
Go version : go1.13.6

Have you tried reproducing the issue with the latest release?

v1.2.4

What is the hardware spec (RAM, OS)?

128G mem & 1.8T SSD

Linux version 3.10.0-1062.9.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) ) #1 SMP Fri Dec 6 15:49:49 UTC 2019

Steps to reproduce the issue (command/config used to run Dgraph).

Just online write with java api, its wierd.

Expected behaviour and actual result.

Alpha crah and cluster is blocked. The log is as followings

`
panic: runtime error: slice bounds out of range [:135467521] with capacity 7968

goroutine 10106572 [running]:
github.com/dgraph-io/badger/v2/table.(*blockIterator).setIdx(0xc001b41120, 0x0)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/iterator.go:89 +0x52c
github.com/dgraph-io/badger/v2/table.(*blockIterator).seekToFirst(...)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/iterator.go:147
github.com/dgraph-io/badger/v2/table.(*Iterator).next(0xc001b41110)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/iterator.go:314 +0x12b
github.com/dgraph-io/badger/v2/table.(*Iterator).next(0xc001b41110)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/iterator.go:323 +0x1b3
github.com/dgraph-io/badger/v2/table.(*Iterator).Next(0xc001b41110)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/iterator.go:379 +0x47
github.com/dgraph-io/badger/v2/table.(*ConcatIterator).Next(0xc682d840a0)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/iterator.go:497 +0x33
github.com/dgraph-io/badger/v2/table.(*node).next(0xc112aec210)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/merge_iterator.go:82 +0x3d
github.com/dgraph-io/badger/v2/table.(*MergeIterator).Next(0xc112aec210)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/merge_iterator.go:158 +0x38
github.com/dgraph-io/badger/v2/table.(*node).next(0xc112aec300)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/merge_iterator.go:80 +0x72
github.com/dgraph-io/badger/v2/table.(*MergeIterator).Next(0xc112aec2c0)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/table/merge_iterator.go:158 +0x38
github.com/dgraph-io/badger/v2.(*Iterator).parseItem(0xc22baf0000, 0x1)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/iterator.go:621 +0x17f
github.com/dgraph-io/badger/v2.(*Iterator).Next(0xc22baf0000)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/iterator.go:566 +0x10b
github.com/dgraph-io/badger/v2.(*Stream).produceKVs.func1(0xc83b156a40, 0x17, 0x20, 0xc83b156a60, 0x17, 0x20, 0x0, 0x0, 0x0)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/stream.go:176 +0xa48
github.com/dgraph-io/badger/v2.(*Stream).produceKVs(0xc5d2ba5ea0, 0x1a971a0, 0xc000038118, 0x0, 0x0)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/stream.go:234 +0x2c6
github.com/dgraph-io/badger/v2.(*Stream).Orchestrate.func1(0xc5d4a7daa0, 0xc5d2ba5ea0, 0x1a971a0, 0xc000038118, 0xc5d4a53ec0)
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/stream.go:337 +0x7f
created by github.com/dgraph-io/badger/v2.(*Stream).Orchestrate
/root/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200515210839-ef28ef36b592/stream.go:334 +0x175
`

[JimWen]

When i shutdown and restart alpha with v1.2.4, the problem is still there. But with v1.2.3, there is no problem.

[jarifibrahim]

I've seen similar failures in badger on PR hypermodeinc/badger#1308 . This is an issue in badger.

[harpy-wings]

we are on heavy load.the server crash every 10 minutes . how we can handle it?
increasing alpha groups could be an option ?

[jarifibrahim]

@h4midr Can you downgrade to v20.03.1? The latest version has this failure. @JimWen you should also switch to v1.2.3 if the crash happens frequently.

We're working on fixing this and we'll do a patch release soon.

[harpy-wings]

my dgraph version is 20.03.2
i recorded my screen and i can share it with you. and i really need to find an option. there is project working on it more than 1.5 years. we grow up with you and now , in real use and the first global use,we screwed. if you interested in records or can help me , i really looking forward to hearing from you.
hassaniyaqoti@gmail.com

[jarifibrahim]

@h4midr Did you try downgrading your version? Dgraph v20.03.1 should work fine.

[danielmai]

@h4midr Have you tried using Dgraph v20.03.1 and are still seeing this issue? We have reached out to you over email as well.

[JimWen]

@h4midr Can you downgrade to v20.03.1? The latest version has this failure. @JimWen you should also switch to v1.2.3 if the crash happens frequently.

We're working on fixing this and we'll do a patch release soon.

yes v1.2.3 is fine

[harpy-wings]

@h4midr Have you tried using Dgraph v20.03.1 and are still seeing this issue? We have reached out to you over email as well.

i'll do it but we are not in situation that have options for test.as soon as exams end i'm going to simulate the process to see how it happens again.
thank you all friends.

[jarifibrahim]

@JimWen @h4midr when does the crash happen? Do you have any logs or any information about when the crash shows up? I'm trying to reproduce the failure.