Dgraph hangs on "Group checksum mismatch for id: 1" error

[makitka2007]

i am trying 1.0.12rc6 on 3 nodes dgraph cluster
first 3 millions were inserted quickly, but after that it stuck on this error on mutations, and doesn't respond on every even very simply query.

[makitka2007]

i see in /state that both 3 checksums are different for 1 group. but what should be done here? all mutations fail. it doesn't seem to be changed. wait infinitely?

[makitka2007]

tried 2 times, but got dgraph hang on this error both times. it was throwing this error all the night, so i have to revert to 1.0.10

[manishrjain]

Can you post the logs, if you have them? And can you list down the steps you took, so we can try reproducing the issue at our end?

[makitka2007]

please find logs for last 15 minutes: https://drive.google.com/file/d/17jLZ7uiWlCUjfjkWuieXJ81mcVPV7ot3/view?usp=sharing

and for time interval, when it started to happen: https://drive.google.com/file/d/11-CLxvHo6voQNXp3ved5aLNuW84raj99/view?usp=sharing

[danielmai]

Can you share the commands you used to run Dgraph? The logs say that the Zero group can't communicate with the leader.

[makitka2007]

node 0
dgraph zero --my=<ip0>:5080 --idx=10 --peer <ip1>:5080 --replicas 3 --logtostderr
dgraph alpha --my=<ip0>:7080 --lru_mb=8192 --idx 1 --zero=<ip0>:5080 --logtostderr --expand_edge=false --pending_proposals=64
node 1
dgraph zero --my=<ip1>:5080 --idx=11 --replicas 3 --logtostderr
dgraph alpha --my=<ip1>:7080 --lru_mb=8192 --idx 2 --zero=<ip1>:5080 --logtostderr --expand_edge=false --pending_proposals=64
node 2
dgraph zero --my=<ip2>:5080 --idx=12 --peer <ip1>:5080 --replicas 3 --logtostderr
dgraph alpha --my=<ip2>:7080 --lru_mb=8192 --idx 3 --zero=<ip2>:5080 --logtostderr --expand_edge=false --pending_proposals=64

IPs are static

[makitka2007]

we have 3 nodes cluster, and each node contains zero and alpha running on it

[manishrjain]

I suspect that one of the Alphas is talking to the wrong Zero for some reason. That Alpha might be the one which is blocking queries. We have code to defend against network partitions, so I'm not entirely sure why this would happen.

I'd recommend restarting the Alpha which has issues.

[makitka2007]

i am not sure that need for manual node restart is a normal case for production..
is there a way to debug which zero address is used by alpha?

actually seems like we have the same problem on 1.0.10 version as well, i think the only difference is that in rc6 checksum checks are introduced, so mutations are failing. for 1.0.10 version we just have 2 nodes working and third one just answering OK for all health checks but mutations and all queries run forever on this node, so all inserts are going to that 2 working nodes.

could you please check start commands if they are correct? instances are started by amazon in random order

[makitka2007]

so, every time we have the same picture - clear cluster starts working, both 3 nodes are asnwering ok and running mutations. but then (after inserting few tens of millions edges) one node hangs and all queries there run infinitely, mutations on this node hang as well

[makitka2007]

okay, i got this again - now i have both 3 nodes being stuck
3 zero nodes saying via /state that leader is xxx.xxx.10.237, but in alpha logs i see:

  | Feb 27 16:40:14.536 | i-xxx860a | dgraph-alpha | Got address of a Zero server: xxx.xxx.9.69:5080
  | Feb 27 16:39:31.648 | i-xxxcb08 | dgraph-alpha | Got address of a Zero server: xxx.xxx.9.69:5080
  | Feb 27 16:39:08.011 | i-xxx09be | dgraph-alpha | Got address of a Zero server: xxx.xxx.9.69:5080
  | Feb 27 16:38:18.023 | i-xxx860a | dgraph-alpha | Got address of a Zero server: xxx.xxx.9.69:5080
  | Feb 27 16:38:17.023 | i-xxx860a | dgraph-alpha | Got address of a Zero server: xxx.xxx.9.69:5080
  | Feb 27 16:38:16.022 | i-xxx860a | dgraph-alpha | Got address of a Zero server: xxx.xxx.9.69:5080

so, all 3 alpha nodes are pointing to wrong zero node

after 3 alphas restart i got following in logs:

  | Feb 27 16:48:20.315 | i-xxx1ba9 | dgraph-alpha | Got address of a Zero server: xxx.xxx.9.69:5080
  | Feb 27 16:47:40.600 | i-xxxcb08 | dgraph-alpha | Got address of a Zero server: xxx.xxx.9.69:5080
  | Feb 27 16:47:30.037 | i-xxx09be | dgraph-alpha | Got address of a Zero server: xxx.xxx.10.237:5080

wheres currently leader is xxx.xxx.8.135 (for both 3 zero nodes state)

[manishrjain]

Hi @makitka2007 , it's hard to debug with snippets of logs. Can you reach out to me via email, and we can dig deeper.

[yashafromrussia]

We saw a similar issue yesterday. Restarting alphas fixed it. It'd be great to see the solution in here. I'll try and post something useful later today