OOM during alpha rebalancing

[selmeci]

If you suspect this could be a bug, follow the template.

• What version of Dgraph are you using?
  v1.0.10

• Have you tried reproducing the issue with latest release?
  Yes

• What is the hardware spec (RAM, OS)?
  amazon m5d.xlarge. 1 zero, 2 alpha

• Steps to reproduce the issue (command/config used to run Dgraph).
  A have lots of data in text.value predicate. When I alter schema with:

curl -X POST localhost:8080/alter -d 'text.value: string @index(term,trigram,nfd) .'

a got error:

I1109 12:04:37.130337       1 predicate_move.go:238] Proposed 4888116 keys. Error: <nil>
I1109 12:09:14.685330       1 index.go:33] Deleting index for text.value
I1109 12:09:20.733521       1 index.go:38] Rebuilding index for text.value
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x130f85d, 0x16)
	/usr/local/go/src/runtime/panic.go:608 +0x72
runtime.sysMap(0xc380000000, 0x4000000, 0x1d29398)
	/usr/local/go/src/runtime/mem_linux.go:156 +0xc7
runtime.(*mheap).sysAlloc(0x1d0fbc0, 0x4000000, 0x7f59203ba130, 0x7f58c3da7d00)
	/usr/local/go/src/runtime/malloc.go:619 +0x1c7
runtime.(*mheap).grow(0x1d0fbc0, 0x16b, 0x0)
	/usr/local/go/src/runtime/mheap.go:920 +0x42
runtime.(*mheap).allocSpanLocked(0x1d0fbc0, 0x16b, 0x1d293a8, 0x20303300000000)
	/usr/local/go/src/runtime/mheap.go:848 +0x337
runtime.(*mheap).alloc_m(0x1d0fbc0, 0x16b, 0x840100, 0x7f590370dd00)
	/usr/local/go/src/runtime/mheap.go:692 +0x119
runtime.(*mheap).alloc.func1()
	/usr/local/go/src/runtime/mheap.go:759 +0x4c
runtime.(*mheap).alloc(0x1d0fbc0, 0x16b, 0x7f58c3010100, 0x844b55)
	/usr/local/go/src/runtime/mheap.go:758 +0x8a
runtime.largeAlloc(0x2d6000, 0x880001, 0x7f59b57f2000)
	/usr/local/go/src/runtime/malloc.go:1019 +0x97
runtime.mallocgc.func1()
	/usr/local/go/src/runtime/malloc.go:914 +0x46
runtime.systemstack(0x0)
	/usr/local/go/src/runtime/asm_amd64.s:351 +0x66
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1229

goroutine 262 [running]:
runtime.systemstack_switch()
	/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc0001a8b18 sp=0xc0001a8b10 pc=0x8894c0
runtime.mallocgc(0x2d6000, 0x12b6960, 0x1d0001, 0xc37fb2a000)
	/usr/local/go/src/runtime/malloc.go:913 +0x896 fp=0xc0001a8bb8 sp=0xc0001a8b18 pc=0x83a8b6
runtime.growslice(0x12b6960, 0xc37fb2a000, 0x48800, 0x48800, 0x48801, 0x0, 0x3a000, 0x48800)
	/usr/local/go/src/runtime/slice.go:204 +0x145 fp=0xc0001a8c20 sp=0xc0001a8bb8 pc=0x872585
github.com/dgraph-io/dgraph/protos/pb.(*Proposal).Unmarshal(0xc0001a8f68, 0xc304bd0000, 0x29d09f6, 0x29d2000, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:12100 +0x566 fp=0xc0001a8da0 sp=0xc0001a8c20 pc=0xcfff76
github.com/dgraph-io/dgraph/worker.(*node).calculateSnapshot(0xc000078f60, 0xa, 0x0, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:935 +0xb1f fp=0xc0001a8ff0 sp=0xc0001a8da0 pc=0xf6ac1f
github.com/dgraph-io/dgraph/worker.(*node).proposeSnapshot(0xc000078f60, 0xa, 0x13, 0xc21c7b16c0)
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:570 +0x4d fp=0xc0001a90d8 sp=0xc0001a8ff0 pc=0xf67acd
github.com/dgraph-io/dgraph/worker.(*node).Run(0xc000078f60)
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:636 +0x426 fp=0xc0001a9fd8 sp=0xc0001a90d8 pc=0xf680f6
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0001a9fe0 sp=0xc0001a9fd8 pc=0x88b5a1
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1090 +0x558

goroutine 1 [semacquire, 33 minutes]:
sync.runtime_Semacquire(0xc0002f20f8)
	/usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc0002f20f0)
	/usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer()
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:340 +0x663
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run()
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:439 +0xa4a
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.init.0.func1(0xc00016e240, 0xc00011e200, 0x0, 0x10)
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:71 +0x52
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).execute(0xc00016e240, 0xc000216300, 0x10, 0x10, 0xc00016e240, 0xc000216300)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:702 +0x2d3
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x1c84980, 0x11, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:783 +0x2dc
github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra.(*Command).Execute(0x1c84980, 0x85fb0d, 0x1c8d20c)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/spf13/cobra/command.go:736 +0x2b
github.com/dgraph-io/dgraph/dgraph/cmd.Execute()
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/root.go:57 +0x36
main.main()
	/go/src/github.com/dgraph-io/dgraph/dgraph/main.go:33 +0x88

goroutine 5 [chan receive, 1 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x1d09900)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/golang/glog/glog.go:882 +0x8b
created by github.com/dgraph-io/dgraph/vendor/github.com/golang/glog.init.0
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/golang/glog/glog.go:410 +0x203

goroutine 34 [syscall, 33 minutes]:
os/signal.signal_recv(0x1347d20)
	/usr/local/go/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:29 +0x41

goroutine 35 [chan receive]:
github.com/dgraph-io/dgraph/x.init.1.func1()
	/go/src/github.com/dgraph-io/dgraph/x/metrics.go:95 +0x7e
created by github.com/dgraph-io/dgraph/x.init.1
	/go/src/github.com/dgraph-io/dgraph/x/metrics.go:90 +0x52c

goroutine 39 [chan receive]:
github.com/dgraph-io/dgraph/worker.(*rateLimiter).bleed(0x1d273f0)
	/go/src/github.com/dgraph-io/dgraph/worker/proposal.go:60 +0x95
created by github.com/dgraph-io/dgraph/worker.init.0
	/go/src/github.com/dgraph-io/dgraph/worker/proposal.go:45 +0x41

goroutine 165 [chan receive]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.(*WaterMark).process(0xc0001920c0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/watermark.go:192 +0x22a
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.(*WaterMark).Init
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/watermark.go:73 +0xbf

goroutine 166 [chan receive, 2 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.(*WaterMark).process(0xc0001921c0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/watermark.go:192 +0x22a
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.(*WaterMark).Init
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/watermark.go:73 +0xbf

goroutine 167 [select, 1 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).updateSize(0xc000051180, 0xc000182180)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:946 +0x125
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:261 +0x7b1

goroutine 168 [select]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00012a070, 0xc000182440)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:248 +0x151
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).startCompact
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:233 +0x82

goroutine 169 [select]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00012a070, 0xc000182440)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:248 +0x151
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).startCompact
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:233 +0x82

goroutine 170 [select]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00012a070, 0xc000182440)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:248 +0x151
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).startCompact
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:233 +0x82

goroutine 171 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).flushMemtable(0xc000051180, 0xc000182460, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:873 +0x179
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:274 +0xe01

goroutine 173 [select, 2 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).doWrites(0xc000051180, 0xc0001825a0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:675 +0x404
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:304 +0xbca

goroutine 174 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*valueLog).waitOnGC(0xc0000512c8, 0xc0001825c0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/value.go:1230 +0x61
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:307 +0xc27

goroutine 175 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.(*WaterMark).process(0xc0000741c0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/watermark.go:192 +0x22a
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.(*WaterMark).Init
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/watermark.go:73 +0xbf

goroutine 176 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.(*WaterMark).process(0xc000074200)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/watermark.go:192 +0x22a
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y.(*WaterMark).Init
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/y/watermark.go:73 +0xbf

goroutine 177 [select, 1 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).updateSize(0xc000512700, 0xc0002a2540)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:946 +0x125
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:261 +0x7b1

goroutine 106 [select]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00012a000, 0xc00013e1a0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:248 +0x151
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).startCompact
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:233 +0x82

goroutine 107 [select]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00012a000, 0xc00013e1a0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:248 +0x151
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).startCompact
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:233 +0x82

goroutine 108 [select]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).runWorker(0xc00012a000, 0xc00013e1a0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:248 +0x151
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*levelsController).startCompact
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/levels.go:233 +0x82

goroutine 109 [chan receive, 6 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).flushMemtable(0xc000512700, 0xc00013e1c0, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:873 +0x179
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:274 +0xe01

goroutine 111 [select, 2 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*DB).doWrites(0xc000512700, 0xc00013e300)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:675 +0x404
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:304 +0xbca

goroutine 112 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.(*valueLog).waitOnGC(0xc000512848, 0xc00013e320)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/value.go:1230 +0x61
created by github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger.Open
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/dgraph-io/badger/db.go:307 +0xc27

goroutine 113 [select, 1 minutes]:
github.com/dgraph-io/dgraph/edgraph.(*ServerState).runVlogGC(0x1d08fa0, 0xc000512700)
	/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:96 +0xfa
created by github.com/dgraph-io/dgraph/edgraph.(*ServerState).initStorage
	/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:176 +0x571

goroutine 194 [select, 1 minutes]:
github.com/dgraph-io/dgraph/edgraph.(*ServerState).runVlogGC(0x1d08fa0, 0xc000051180)
	/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:96 +0xfa
created by github.com/dgraph-io/dgraph/edgraph.(*ServerState).initStorage
	/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:177 +0x59f

goroutine 195 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/edgraph.(*ServerState).fillTimestampRequests(0x1d08fa0)
	/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:203 +0x3a7
created by github.com/dgraph-io/dgraph/edgraph.InitServerState
	/go/src/github.com/dgraph-io/dgraph/edgraph/server.go:78 +0xf7

goroutine 196 [chan receive]:
github.com/dgraph-io/dgraph/posting.(*listCache).removeOldestLoop(0xc000198500)
	/go/src/github.com/dgraph-io/dgraph/posting/lru.go:114 +0x95
created by github.com/dgraph-io/dgraph/posting.newListCache
	/go/src/github.com/dgraph-io/dgraph/posting/lru.go:67 +0xed

goroutine 197 [select]:
github.com/dgraph-io/dgraph/posting.periodicUpdateStats(0xc00013e3a0)
	/go/src/github.com/dgraph-io/dgraph/posting/lists.go:126 +0x15c
created by github.com/dgraph-io/dgraph/posting.Init
	/go/src/github.com/dgraph-io/dgraph/posting/lists.go:220 +0xb3

goroutine 198 [select, 1 minutes]:
github.com/dgraph-io/dgraph/posting.updateMemoryMetrics(0xc00013e3a0)
	/go/src/github.com/dgraph-io/dgraph/posting/lists.go:182 +0x1b3
created by github.com/dgraph-io/dgraph/posting.Init
	/go/src/github.com/dgraph-io/dgraph/posting/lists.go:221 +0xd7

goroutine 179 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run.func4(0xc000171b60, 0xc00034c200)
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:417 +0x41
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.run
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:414 +0xa16

goroutine 266 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/worker.(*node).Run.func1(0xc000078f60, 0xc0000917a0)
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:597 +0x3f
created by github.com/dgraph-io/dgraph/worker.(*node).Run
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:596 +0x179

goroutine 181 [IO wait, 33 minutes]:
internal/poll.runtime_pollWait(0x7f59b5796e30, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc000194118, 0x72, 0xc000192600, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc000194118, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc000194100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc000194100, 0xc0001420b0, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000186010, 0xc00031de28, 0xc00031de30, 0x18)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc000186010, 0x13472c0, 0xc000515c80, 0x142dd60, 0xc0001420b0)
	/usr/local/go/src/net/tcpsock.go:260 +0x47
google.golang.org/grpc.(*Server).Serve(0xc000515c80, 0x1429000, 0xc000186010, 0x0, 0x0)
	/go/src/google.golang.org/grpc/server.go:556 +0x210
github.com/dgraph-io/dgraph/worker.RunServer(0xc000000301)
	/go/src/github.com/dgraph-io/dgraph/worker/worker.go:84 +0x2bc
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:278 +0x55

goroutine 226 [IO wait, 33 minutes]:
internal/poll.runtime_pollWait(0x7f59b5796d60, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00036a218, 0x72, 0xc00016c700, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00036a218, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc00036a200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc00036a200, 0x4c00000000000010, 0xc0002bd598, 0x4c6c6355fdac6ba7)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000142098, 0x8597c2, 0xc00016c770, 0xc0002bd628)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc000142098, 0x13472b8, 0xc0001cf980, 0xc000182420, 0x0)
	/usr/local/go/src/net/tcpsock.go:260 +0x47
google.golang.org/grpc.(*Server).Serve(0xc0001cf980, 0x1429000, 0xc000142098, 0x0, 0x0)
	/go/src/google.golang.org/grpc/server.go:556 +0x210
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.serveGRPC(0x1429000, 0xc000142098, 0x0, 0xc0002f20f0)
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:249 +0x21d
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:327 +0x48d

goroutine 227 [IO wait]:
internal/poll.runtime_pollWait(0x7f59b5796f00, 0x72, 0x0)
	/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00036a118, 0x72, 0xc000192800, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00036a118, 0xffffffffffffff00, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc00036a100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc00036a100, 0x819cc5e0, 0xea8010fe0ff4fb8f, 0x87aff8)
	/usr/local/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc000142090, 0x5be5796c, 0xc000321e20, 0x8adce6)
	/usr/local/go/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc000142090, 0xc000321e70, 0x18, 0xc0001cf680, 0xad4485)
	/usr/local/go/src/net/tcpsock.go:260 +0x47
net/http.(*Server).Serve(0xc00015b1e0, 0x1429000, 0xc000142090, 0x0, 0x0)
	/usr/local/go/src/net/http/server.go:2826 +0x22f
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.serveHTTP(0x1429000, 0xc000142090, 0x0, 0xc0002f20f0)
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:267 +0x2da
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:328 +0x4d9

goroutine 228 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer.func1(0xc0002f20f0, 0x1429000, 0xc000142098, 0x1429000, 0xc000142090)
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:332 +0x60
created by github.com/dgraph-io/dgraph/dgraph/cmd/alpha.setupServer
	/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/alpha/run.go:330 +0x532

goroutine 42 [select, 33 minutes]:
google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc000270180)
	/go/src/google.golang.org/grpc/resolver_conn_wrapper.go:111 +0x142
created by google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/google.golang.org/grpc/resolver_conn_wrapper.go:97 +0x3f

goroutine 43 [select, 33 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0002703c0)
	/go/src/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x10f
created by google.golang.org/grpc.newCCBalancerWrapper
	/go/src/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x14e

goroutine 242 [IO wait]:
internal/poll.runtime_pollWait(0x7f59b5796c90, 0x72, 0xc000526ba8)
	/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00011a198, 0x72, 0xffffffffffffff00, 0x141d300, 0x1c8d5b8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00011a198, 0xc00059a000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00011a180, 0xc00059a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00011a180, 0xc00059a000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00034e008, 0xc00059a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc00017c660, 0xc00022c038, 0x9, 0x9, 0x20, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x141b280, 0xc00017c660, 0xc00022c038, 0x9, 0x9, 0x9, 0xc07f833800, 0x82fede, 0xc000526e8b)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(0x141b280, 0xc00017c660, 0xc00022c038, 0x9, 0x9, 0x392e7f38, 0x392e7f381bc8412c, 0x5be57965)
	/usr/local/go/src/io/io.go:329 +0x58
golang.org/x/net/http2.readFrameHeader(0xc00022c038, 0x9, 0x9, 0x141b280, 0xc00017c660, 0x0, 0xbef17c3900000000, 0x1cf56c311b0, 0x1d091e0)
	/go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00022c000, 0xc01bc84120, 0xc01bc84120, 0x0, 0x0)
	/go/src/golang.org/x/net/http2/frame.go:492 +0xa3
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000144380)
	/go/src/google.golang.org/grpc/internal/transport/http2_client.go:1218 +0x176
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xc53

goroutine 243 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000743c0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000308960, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000144380)
	/go/src/google.golang.org/grpc/internal/transport/http2_client.go:328 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/google.golang.org/grpc/internal/transport/http2_client.go:326 +0xdae

goroutine 182 [select, 33 minutes]:
google.golang.org/grpc.newClientStream.func5(0xc00027c000, 0xc000164000, 0x1429dc0, 0xc00018f500)
	/go/src/google.golang.org/grpc/stream.go:311 +0xd7
created by google.golang.org/grpc.newClientStream
	/go/src/google.golang.org/grpc/stream.go:310 +0x887

goroutine 246 [chan receive]:
github.com/dgraph-io/dgraph/conn.(*Pool).MonitorHealth(0xc000140230)
	/go/src/github.com/dgraph-io/dgraph/conn/pool.go:186 +0x67
created by github.com/dgraph-io/dgraph/conn.NewPool
	/go/src/github.com/dgraph-io/dgraph/conn/pool.go:140 +0x30d

goroutine 252 [select, 33 minutes]:
google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc000074dc0)
	/go/src/google.golang.org/grpc/resolver_conn_wrapper.go:111 +0x142
created by google.golang.org/grpc.(*ccResolverWrapper).start
	/go/src/google.golang.org/grpc/resolver_conn_wrapper.go:97 +0x3f

goroutine 248 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000074980, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000078a20, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00012e600)
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xe93

goroutine 249 [select, 33 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00012e600)
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xeb8

goroutine 250 [IO wait]:
internal/poll.runtime_pollWait(0x7f59b5796bc0, 0x72, 0xc000522b58)
	/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00014c018, 0x72, 0xffffffffffffff00, 0x141d300, 0x1c8d5b8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00014c018, 0xc0005c2000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00014c000, 0xc0005c2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00014c000, 0xc0005c2000, 0x8000, 0x8000, 0x2f, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00034e048, 0xc0005c2000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc00017c8a0, 0xc00022c1f8, 0x9, 0x9, 0x18, 0x1232380, 0x0)
	/usr/local/go/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x141b280, 0xc00017c8a0, 0xc00022c1f8, 0x9, 0x9, 0x9, 0x83a30f, 0x82fede, 0xc000522da0)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(0x141b280, 0xc00017c8a0, 0xc00022c1f8, 0x9, 0x9, 0x6, 0xc00003ea88, 0xc00003ea00)
	/usr/local/go/src/io/io.go:329 +0x58
golang.org/x/net/http2.readFrameHeader(0xc00022c1f8, 0x9, 0x9, 0x141b280, 0xc00017c8a0, 0x0, 0xc000000000, 0xc000522e38, 0xc000522eb0)
	/go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00022c1c0, 0xc01bc84160, 0xc01bc84160, 0x0, 0x0)
	/go/src/golang.org/x/net/http2/frame.go:492 +0xa3
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00012e600, 0xc00031b380, 0x1347318)
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
google.golang.org/grpc.(*Server).serveStreams(0xc000515c80, 0x142ecc0, 0xc00012e600)
	/go/src/google.golang.org/grpc/server.go:681 +0xdd
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000515c80, 0x142ecc0, 0xc00012e600)
	/go/src/google.golang.org/grpc/server.go:643 +0x43
created by google.golang.org/grpc.(*Server).handleRawConn
	/go/src/google.golang.org/grpc/server.go:642 +0x57e

goroutine 235 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00016cc00, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00013c840, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00ae56000)
	/go/src/google.golang.org/grpc/internal/transport/http2_client.go:328 +0x7b
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/google.golang.org/grpc/internal/transport/http2_client.go:326 +0xdae

goroutine 230 [select]:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00016c9c0, 0x1, 0x0, 0x0, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x103
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000170e40, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1ac
google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0001cfc80)
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xe93

goroutine 231 [select, 33 minutes]:
google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0001cfc80)
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x1e4
created by google.golang.org/grpc/internal/transport.newHTTP2Server
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xeb8

goroutine 232 [IO wait]:
internal/poll.runtime_pollWait(0x7f59b5796af0, 0x72, 0xc000060b58)
	/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00036a518, 0x72, 0xffffffffffffff00, 0x141d300, 0x1c8d5b8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00036a518, 0xc00ae0a000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00036a500, 0xc00ae0a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00036a500, 0xc00ae0a000, 0x8000, 0x8000, 0x2f, 0x0, 0x200)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001420b0, 0xc00ae0a000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc000078b40, 0xc00ae26038, 0x9, 0x9, 0x18, 0x1232380, 0x0)
	/usr/local/go/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x141b280, 0xc000078b40, 0xc00ae26038, 0x9, 0x9, 0x9, 0x83a30f, 0x82fede, 0xc000060da0)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(0x141b280, 0xc000078b40, 0xc00ae26038, 0x9, 0x9, 0x6, 0xc000390088, 0xc000390000)
	/usr/local/go/src/io/io.go:329 +0x58
golang.org/x/net/http2.readFrameHeader(0xc00ae26038, 0x9, 0x9, 0x141b280, 0xc000078b40, 0x0, 0xc100000000, 0xc000060e38, 0xc000060eb0)
	/go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00ae26000, 0xc015698020, 0xc015698020, 0x0, 0x0)
	/go/src/golang.org/x/net/http2/frame.go:492 +0xa3
google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0001cfc80, 0xc00ae2c030, 0x1347318)
	/go/src/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c
google.golang.org/grpc.(*Server).serveStreams(0xc000515c80, 0x142ecc0, 0xc0001cfc80)
	/go/src/google.golang.org/grpc/server.go:681 +0xdd
google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000515c80, 0x142ecc0, 0xc0001cfc80)
	/go/src/google.golang.org/grpc/server.go:643 +0x43
created by google.golang.org/grpc.(*Server).handleRawConn
	/go/src/google.golang.org/grpc/server.go:642 +0x57e

goroutine 253 [select, 33 minutes]:
google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000074fc0)
	/go/src/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x10f
created by google.golang.org/grpc.newCCBalancerWrapper
	/go/src/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x14e

goroutine 241 [chan receive, 2 minutes]:
github.com/dgraph-io/dgraph/x.(*WaterMark).process(0xc00011e4c0)
	/go/src/github.com/dgraph-io/dgraph/x/watermark.go:188 +0x22a
created by github.com/dgraph-io/dgraph/x.(*WaterMark).Init
	/go/src/github.com/dgraph-io/dgraph/x/watermark.go:70 +0xbe

goroutine 234 [IO wait]:
internal/poll.runtime_pollWait(0x7f59b5796a20, 0x72, 0xc0005beba8)
	/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00014c298, 0x72, 0xffffffffffffff00, 0x141d300, 0x1c8d5b8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00014c298, 0xc00ae3e000, 0x8000, 0x8000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00014c280, 0xc00ae3e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00014c280, 0xc00ae3e000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0001420c8, 0xc00ae3e000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x68
bufio.(*Reader).Read(0xc000078d20, 0xc00ae261f8, 0x9, 0x9, 0x20, 0x0, 0x0)
	/usr/local/go/src/bufio/bufio.go:216 +0x22f
io.ReadAtLeast(0x141b280, 0xc000078d20, 0xc00ae261f8, 0x9, 0x9, 0x9, 0xc0a276ae10, 0x82fede, 0xc0005bee8b)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(0x141b280, 0xc000078d20, 0xc00ae261f8, 0x9, 0x9, 0x3af8ffc3, 0x3af8ffc31bc841ec, 0x5be57965)
	/usr/local/go/src/io/io.go:329 +0x58
golang.org/x/net/http2.readFrameHeader(0xc00ae261f8, 0x9, 0x9, 0x141b280, 0xc000078d20, 0x0, 0xbef17c3900000000, 0x1cf588e1f30, 0x1d091e0)
	/go/src/golang.org/x/net/http2/frame.go:237 +0x7b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc00ae261c0, 0xc01bc841e0, 0xc01bc841e0, 0x0, 0x0)
	/go/src/golang.org/x/net/http2/frame.go:492 +0xa3
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00ae56000)
	/go/src/google.golang.org/grpc/internal/transport/http2_client.go:1218 +0x176
created by google.golang.org/grpc/internal/transport.newHTTP2Client
	/go/src/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xc53

goroutine 237 [chan receive]:
github.com/dgraph-io/dgraph/conn.(*Pool).MonitorHealth(0xc00ae25a90)
	/go/src/github.com/dgraph-io/dgraph/conn/pool.go:186 +0x67
created by github.com/dgraph-io/dgraph/conn.NewPool
	/go/src/github.com/dgraph-io/dgraph/conn/pool.go:140 +0x30d

goroutine 258 [select]:
github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft.(*node).run(0xc000079020, 0xc00018a3c0)
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft/node.go:313 +0x4dc
created by github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft.StartNode
	/go/src/github.com/dgraph-io/dgraph/vendor/github.com/coreos/etcd/raft/node.go:210 +0x41d

goroutine 259 [select, 3 minutes]:
github.com/dgraph-io/dgraph/worker.(*node).processRollups(0xc000078f60)
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:344 +0x180
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1087 +0x4e6

goroutine 260 [running]:
	goroutine running on other thread; stack unavailable
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1088 +0x50b

goroutine 261 [chan receive, 33 minutes]:
github.com/dgraph-io/dgraph/conn.(*Node).BatchAndSendMessages(0xc00011e400)
	/go/src/github.com/dgraph-io/dgraph/conn/node.go:308 +0x722
created by github.com/dgraph-io/dgraph/worker.(*node).InitAndStartNode
	/go/src/github.com/dgraph-io/dgraph/worker/draft.go:1089 +0x533

goroutine 263 [select, 2 minutes]:
github.com/dgraph-io/dgraph/worker.(*groupi).periodicMembershipUpdate(0xc00036a080)
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:553 +0x185
created by github.com/dgraph-io/dgraph/worker.StartRaftNodes
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:142 +0x7b3

goroutine 265 [select, 2 minutes]:
github.com/dgraph-io/dgraph/worker.(*groupi).processOracleDeltaStream.func1()
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:773 +0x8e8
github.com/dgraph-io/dgraph/worker.(*groupi).processOracleDeltaStream(0xc00036a080)
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:841 +0x177
created by github.com/dgraph-io/dgraph/worker.StartRaftNodes
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:144 +0x7fb

goroutine 183 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000272640, 0xc00013e5f0, 0x5, 0x5, 0x12b3ec0, 0x7f59b57f3b00, 0x100000000000000)
	/go/src/google.golang.org/grpc/internal/transport/transport.go:142 +0x1ae
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000272640, 0xc00013e5f0, 0x5, 0x5, 0x7f59b57f3b00, 0x0, 0xc0005ba901)
	/go/src/google.golang.org/grpc/internal/transport/transport.go:131 +0x5a
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00018f650, 0xc00013e5f0, 0x5, 0x5, 0xc0005baa01, 0xc0005ba9b0, 0xc280a2)
	/go/src/google.golang.org/grpc/internal/transport/transport.go:414 +0x55
io.ReadAtLeast(0x141bda0, 0xc00018f650, 0xc00013e5f0, 0x5, 0x5, 0x5, 0xc000144380, 0xc00af1e000, 0x5)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(0x141bda0, 0xc00018f650, 0xc00013e5f0, 0x5, 0x5, 0x0, 0x0, 0x0)
	/usr/local/go/src/io/io.go:329 +0x58
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00af1e000, 0xc00013e5f0, 0x5, 0x5, 0x0, 0x0, 0x0)
	/go/src/google.golang.org/grpc/internal/transport/transport.go:398 +0xbf
google.golang.org/grpc.(*parser).recvMsg(0xc00013e5e0, 0x100000000, 0xc0005babe8, 0x83a48c, 0xc026b23500, 0xc142794000, 0x38000, 0x1261dc0)
	/go/src/google.golang.org/grpc/rpc_util.go:476 +0x65
google.golang.org/grpc.recvAndDecompress(0xc00013e5e0, 0xc00af1e000, 0x0, 0x0, 0x100000000, 0x0, 0x0, 0x0, 0xc1427adcc0, 0x4, ...)
	/go/src/google.golang.org/grpc/rpc_util.go:607 +0x4d
google.golang.org/grpc.recv(0xc00013e5e0, 0x7f59b575e360, 0x1d27630, 0xc00af1e000, 0x0, 0x0, 0x12c18c0, 0xc1424ca000, 0x100000000, 0x0, ...)
	/go/src/google.golang.org/grpc/rpc_util.go:650 +0x9b
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00ae60160, 0x12c18c0, 0xc1424ca000, 0x0, 0x0, 0xc1424ca000)
	/go/src/google.golang.org/grpc/stream.go:860 +0x106
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc00ae60160, 0x0, 0x203050)
	/go/src/google.golang.org/grpc/stream.go:711 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc000164000, 0xc0005baea0, 0xc0005bae90, 0x1000008000104, 0x60)
	/go/src/google.golang.org/grpc/stream.go:563 +0x281
google.golang.org/grpc.(*clientStream).RecvMsg(0xc000164000, 0x12c18c0, 0xc1424ca000, 0x4000106, 0x0)
	/go/src/google.golang.org/grpc/stream.go:710 +0x116
github.com/dgraph-io/dgraph/protos/pb.(*zeroUpdateClient).Recv(0xc0002a0290, 0xc047916a00, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:3752 +0x62
github.com/dgraph-io/dgraph/worker.(*groupi).periodicMembershipUpdate.func1(0x142dbe0, 0xc0002a0290, 0x1429d00, 0xc0001932c0, 0xc0002a0250, 0xc00036a080)
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:533 +0x4e
created by github.com/dgraph-io/dgraph/worker.(*groupi).periodicMembershipUpdate
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:530 +0x53c

goroutine 211 [select, 33 minutes]:
google.golang.org/grpc.newClientStream.func5(0xc00027c000, 0xc00013a240, 0x1429dc0, 0xc00af2c570)
	/go/src/google.golang.org/grpc/stream.go:311 +0xd7
created by google.golang.org/grpc.newClientStream
	/go/src/google.golang.org/grpc/stream.go:310 +0x887

goroutine 212 [select, 2 minutes]:
google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc00b377630, 0xc00af54390, 0x5, 0x5, 0xc00af4f8f0, 0xc00af4f8f0, 0x86c5bd)
	/go/src/google.golang.org/grpc/internal/transport/transport.go:142 +0x1ae
google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00b377630, 0xc00af54390, 0x5, 0x5, 0x2, 0xc00af4fa8c, 0x2)
	/go/src/google.golang.org/grpc/internal/transport/transport.go:131 +0x5a
google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00af2c660, 0xc00af54390, 0x5, 0x5, 0x1, 0xc00af4f990, 0xc280a2)
	/go/src/google.golang.org/grpc/internal/transport/transport.go:414 +0x55
io.ReadAtLeast(0x141bda0, 0xc00af2c660, 0xc00af54390, 0x5, 0x5, 0x5, 0xc000144380, 0xc00b3e8000, 0x5)
	/usr/local/go/src/io/io.go:310 +0x88
io.ReadFull(0x141bda0, 0xc00af2c660, 0xc00af54390, 0x5, 0x5, 0xc00af4fa50, 0x92192c, 0xc00af2c570)
	/usr/local/go/src/io/io.go:329 +0x58
google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00b3e8000, 0xc00af54390, 0x5, 0x5, 0x1, 0x0, 0x100000001)
	/go/src/google.golang.org/grpc/internal/transport/transport.go:398 +0xbf
google.golang.org/grpc.(*parser).recvMsg(0xc00af54380, 0x100000000, 0xc0d319e610, 0x3, 0x3, 0x0, 0x0, 0x1426f40)
	/go/src/google.golang.org/grpc/rpc_util.go:476 +0x65
google.golang.org/grpc.recvAndDecompress(0xc00af54380, 0xc00b3e8000, 0x0, 0x0, 0x100000000, 0x0, 0x0, 0x0, 0xc0d319e610, 0x3, ...)
	/go/src/google.golang.org/grpc/rpc_util.go:607 +0x4d
google.golang.org/grpc.recv(0xc00af54380, 0x7f59b575e360, 0x1d27630, 0xc00b3e8000, 0x0, 0x0, 0x12ac9a0, 0xc1ea075b80, 0x100000000, 0x0, ...)
	/go/src/google.golang.org/grpc/rpc_util.go:650 +0x9b
google.golang.org/grpc.(*csAttempt).recvMsg(0xc00b3e60b0, 0x12ac9a0, 0xc1ea075b80, 0x0, 0x3, 0xc1ea075b80)
	/go/src/google.golang.org/grpc/stream.go:860 +0x106
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc00b3e60b0, 0xc00af58420, 0x20307a)
	/go/src/google.golang.org/grpc/stream.go:711 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc00013a240, 0xc00af4fe80, 0xc00af4fe70, 0x0, 0x28)
	/go/src/google.golang.org/grpc/stream.go:563 +0x281
google.golang.org/grpc.(*clientStream).RecvMsg(0xc00013a240, 0x12ac9a0, 0xc1ea075b80, 0x0, 0xc00b3e4060)
	/go/src/google.golang.org/grpc/stream.go:710 +0x116
github.com/dgraph-io/dgraph/protos/pb.(*zeroOracleClient).Recv(0xc00af2e090, 0xc00af4ff38, 0x2, 0x0)
	/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:3784 +0x62
github.com/dgraph-io/dgraph/worker.(*groupi).processOracleDeltaStream.func1.1(0x142d040, 0xc00af2e090, 0xc00af58420, 0x1429d00, 0xc00af56100)
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:756 +0xac
created by github.com/dgraph-io/dgraph/worker.(*groupi).processOracleDeltaStream.func1
	/go/src/github.com/dgraph-io/dgraph/worker/groups.go:749 +0x342

goroutine 39086258 [IO wait]:
internal/poll.runtime_pollWait(0x7f59b5796950, 0x72, 0xc1c8705b90)
	/usr/local/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc1c8030018, 0x72, 0xffffffffffffff00, 0x141d300, 0x1c8d5b8)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc1c8030018, 0xc1c86cc000, 0x1000, 0x1000)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc1c8030000, 0xc1c86cc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc1c8030000, 0xc1c86cc000, 0x1000, 0x1000, 0x94790b, 0xc1c8705ca0, 0xacabaa)
	/usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc1c8034000, 0xc1c86cc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/local/go/src/net/net.go:177 +0x68
net/http.(*connReader).Read(0xc1c8032090, 0xc1c86cc000, 0x1000, 0x1000, 0xbef17c58ee285abc, 0x1eca1f2eb3d, 0x1d091e0)
	/usr/local/go/src/net/http/server.go:786 +0xfc
bufio.(*Reader).fill(0xc09dc0c000)
	/usr/local/go/src/bufio/bufio.go:100 +0x106
bufio.(*Reader).Peek(0xc09dc0c000, 0x4, 0x1eca1f2eb3d, 0x1d091e0, 0x0, 0x0, 0x1d091e0)
	/usr/local/go/src/bufio/bufio.go:132 +0x3f
net/http.(*conn).serve(0xc1c8036000, 0x1429d00, 0xc1c8038000)
	/usr/local/go/src/net/http/server.go:1872 +0x7ba
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2851 +0x2f5

goroutine 39086621 [select, 2 minutes]:
github.com/dgraph-io/dgraph/worker.(*node).proposeAndWait.func1(0x1176592e000, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/worker/proposal.go:177 +0x4c6
github.com/dgraph-io/dgraph/worker.(*node).proposeAndWait(0xc000078f60, 0x1429dc0, 0xc1fa64c270, 0xc1c8030b00, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/worker/proposal.go:201 +0x6b6
github.com/dgraph-io/dgraph/worker.(*grpcWorker).Mutate(0xc00034c228, 0x1429dc0, 0xc1fa64c270, 0xc06f6e8070, 0x0, 0x0, 0x0)
	/go/src/github.com/dgraph-io/dgraph/worker/mutation.go:589 +0x14a
github.com/dgraph-io/dgraph/protos/pb._Worker_Mutate_Handler(0x1291ac0, 0xc00034c228, 0x1429dc0, 0xc1fa64c270, 0xc230da4370, 0x0, 0x0, 0x0, 0xc204fa8060, 0x2b)
	/go/src/github.com/dgraph-io/dgraph/protos/pb/pb.pb.go:4227 +0x23e
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000515c80, 0x142ecc0, 0xc0001cfc80, 0xc11151d000, 0xc00018eea0, 0x1c83280, 0x0, 0x0, 0x0)
	/go/src/google.golang.org/grpc/server.go:966 +0x4a2
google.golang.org/grpc.(*Server).handleStream(0xc000515c80, 0x142ecc0, 0xc0001cfc80, 0xc11151d000, 0x0)
	/go/src/google.golang.org/grpc/server.go:1245 +0xd58
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002f21d0, 0xc000515c80, 0x142ecc0, 0xc0001cfc80, 0xc11151d000)
	/go/src/google.golang.org/grpc/server.go:685 +0x9f
created by google.golang.org/grpc.(*Server).serveStreams.func1
	/go/src/google.golang.org/grpc/server.go:683 +0xa1

• Expected behaviour and actual result.
  dGraph is running and using tokenizer

This is tokenizer code:

package main

import (
	"unicode"
	"strings"
    "golang.org/x/text/transform"
    "golang.org/x/text/unicode/norm"
)

func Tokenizer() interface{} { return NfdTokenizer{} }

type NfdTokenizer struct{}

func (NfdTokenizer) Name() string     { return "nfd" }
func (NfdTokenizer) Type() string     { return "string" }
func (NfdTokenizer) Identifier() byte { return 0x81 }

func isMn(r rune) bool {
    return unicode.Is(unicode.Mn, r) // Mn: nonspacing marks
}

func (t NfdTokenizer) Tokens(value interface{}) ([]string, error) {
	var toks []string
	var lowerText string
	lowerText = strings.ToLower(value.(string))
	tr := transform.Chain(norm.NFD, transform.RemoveFunc(isMn), norm.NFC)
	result, _, _ := transform.String(tr, lowerText)
	toks = strings.Fields(result)
	return toks, nil
}

[selmeci]

After restart I can see:
E1109 12:26:23.171312 1 groups.go:826] While proposing delta: max_assigned:10170 . Error=Server unavailable. Please retry later. Retrying...

[manishrjain]

The server unavailable just means the server is exceeding pending proposals limit. Hard to say about the OOM. You'd need to give more details about the heap and such.

Doesn't seem like a bug report. This can be moved to discuss.dgraph.io.

[selmeci]

Ok, I can move it. Maybe I can prevent this error when I use tokenizer during bulk load. Is it possible?

[manishrjain]

First thing to do would be to run a heap profiler every minute or so, so you can catch the profile just before a crash. If you look at that, it would help identify the memory issue.

What I can see in your tokenizer is that you're creating a transformer for every call, that's something you can just create once and reuse. Maybe it helps, just maybe...

[selmeci]

It is not about custom tokenizer. My cluster crashes every time during rebalancing after I start second alpha.

[selmeci]

Same problem was reported here: #2638

[manishrjain]

Without a heap profile, it's hard to comment on this. Dgraph spawns 16 goroutines to pick up posting lists from disk, marshal them and send them over the wire. It's possible that your instance doesn't have enough memory to support this.

[selmeci]

I don't think so. Everything was working in v1.0.8 (even custom tokenizer). This is my command for cluster:

docker run -d --name=zero --hostname=1.zero.weave.local --restart unless-stopped -v /dgraph/zero/w:/dgraph/zw -p 6080:6080 dgraph/dgraph:v1.0.10 dgraph zero --my=1.zero.weave.local:5080 --idx 1 --replicas 1 --bindall=true -w /dgraph/zw

docker run -d --name=alpha --hostname=1.alpha.weave.local --restart unless-stopped -v /dgraph/export:/dgraph/export -v /dgraph/alpha/w:/dgraph/w  -v /dgraph/alpha/p:/dgraph/p -p 8080:8080 dgraph/dgraph:v1.0.10 dgraph alpha --export=/dgraph/export --my=1.alpha.weave.local:7080 --lru_mb 4096 --zero zero.weave.local:5080 -p /dgraph/p --idx=1 -w /dgraph/w --badger.vlog=disk --whitelist 172.17.0.0:172.20.0.0 --bindall=true

docker run -d --name=alpha --hostname=2.alpha.weave.local --restart unless-stopped -v /dgraph/export:/dgraph/export -v /dgraph/alpha/w:/dgraph/w  -v /dgraph/alpha/p:/dgraph/p -p 8080:8080 dgraph/dgraph:v1.0.10 dgraph alpha --export=/dgraph/export --my=2.alpha.weave.local:7080 --lru_mb 4096 --zero zero.weave.local:5080 -p /dgraph/p --idx=2 -w /dgraph/w --badger.vlog=disk --whitelist 172.17.0.0:172.20.0.0 --bindall=true

During rebalancing several predicates are moved from alpha 1 to alpha 2 and RAM consumption on second alpha is only increasing and never is released. I attached heap profiles.

Second problem is, that after I alter schema:

<text.value>: string @index(term, trigram, fulltext) .
<text.value>: string @index(term, trigram) .

I cannot find anything in this predicate and this logs appear:

1 index.go:33] Deleting index for text.value
I1112 10:06:11.385218       1 index.go:38] Rebuilding index for text.value
E1112 10:11:46.399512       1 groups.go:826] While proposing delta: max_assigned:10009 . Error=Server unavailable. Please retry later. Retrying...
E1112 10:12:14.408684       1 groups.go:826] While proposing delta: max_assigned:10009 . Error=Server unavailable. Please retry later. Retrying...
E1112 10:12:42.409297       1 groups.go:826] While proposing delta: max_assigned:10009 . Error=Server unavailable. Please retry later. Retrying...
E1112 10:13:10.411981       1 groups.go:826] While proposing delta: max_assigned:10009 . Error=Server unavailable. Please retry later. Retrying...
E1112 10:13:38.412603       1 groups.go:826] While proposing delta: max_assigned:10009 . Error=Server unavailable. Please retry later. Retrying...
E1112 10:14:06.642752       1 groups.go:826] While proposing delta: max_assigned:10009 . Error=Server unavailable. Please retry later. Retrying...
E1112 10:14:34.643410       1 groups.go:826] While proposing delta: max_assigned:10009 . Error=Server unavailable. Please retry later. Retrying...

pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.002.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.003.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.004.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.005.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.006.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.007.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.008.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.009.pb.gz
pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.010.pb.gz

[selmeci]

After while I can see this:

Applying proposal. Error: context deadline exceeded. Proposal: "mutations:<group_id:2 start_ts:10008 schema:<predicate:\"text.value\" value_type:STRING directive:INDEX tokenizer:\"term\" tokenizer:\"trigram\" tokenizer:\"fulltext\" > > key:\"02-2932068803782637718\" ".

[manishrjain]

During index rebuild, it if is going to take time, other proposals are going to be blocked. That's why you're seeing those errors.

If you're on Slack, ping Daniel Mai, and we can help you get a heap profile for us. That's the best we can do to identify the OOM issue. Without that profile, we can't really help you.

[manishrjain]

Also, can you share the full logs? Run them with -v=2 --logtostderr?

[selmeci]

Hi, I attached several heap profiles in my previous post (at the end). Here are again:

https://github.com/dgraph-io/dgraph/files/2571496/pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.010.pb.gz

https://github.com/dgraph-io/dgraph/files/2571495/pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.009.pb.gz

https://github.com/dgraph-io/dgraph/files/2571494/pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.008.pb.gz

....

I turned off this cluster for now and continue with v1.0.8 but I can start it again and download full logs during this week.