Cluster fails to converge on startup

[aphyr]

On the current (2018/02/17) nightlies, when setting up a cluster of five nodes, all running dgraph zero and alpha, some alpha nodes get stuck loading snapshots and return "server not ready" errors for requests indefinitely. In 20180217T143148.000-0600.zip, for instance, node n1 repeatedly tries to fetch a snapshot, but never succeeds:

2018/02/17 12:32:08 draft.go:139: Node ID: 1 with GroupID: 1
2018/02/17 12:32:08 node.go:258: Group 1 found 0 entries
2018/02/17 12:32:08 draft.go:673: New Node for group: 1
2018/02/17 12:32:08 draft.go:678: Retrieving snapshot.
2018/02/17 12:32:08 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Group 1 not served.
. Retrying...
2018/02/17 12:32:09 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
2018/02/17 12:32:10 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
2018/02/17 12:32:11 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
2018/02/17 12:32:12 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
2018/02/17 12:32:13 predicate.go:51: Getting SNAPSHOT: Time elapsed: 05s, bytes written: 0 B
2018/02/17 12:32:13 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
2018/02/17 12:32:14 predicate.go:51: Getting SNAPSHOT: Time elapsed: 05s, bytes written: 0 B
2018/02/17 12:32:14 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
2018/02/17 12:32:15 predicate.go:51: Getting SNAPSHOT: Time elapsed: 05s, bytes written: 0 B
2018/02/17 12:32:15 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
2018/02/17 12:32:16 predicate.go:51: Getting SNAPSHOT: Time elapsed: 05s, bytes written: 0 B
2018/02/17 12:32:16 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
2018/02/17 12:32:17 predicate.go:51: Getting SNAPSHOT: Time elapsed: 05s, bytes written: 0 B
2018/02/17 12:32:17 draft.go:649: Error while calling fn: Cannot retrieve snapshot from peer, error: rpc error: code = Unknown desc = Not leader of group: 1
. Retrying...
...

[pawanrawal]

This was fixed by 5563bd2. We are going to do a v1.0.4 release this week and the fix would be part of the release.