Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix deadlock in 10-node cluster convergence #2467

Merged
merged 8 commits into from Jul 1, 2018
Merged

Conversation

manishrjain
Copy link
Contributor

@manishrjain manishrjain commented Jul 1, 2018

This PR fixes #2286 .

  • CheckQuorum was causing us multiple issues. When doing a 5-node Zero cluster
    bootstrap, it would cause a leader to step down when the size of the cluster
    is 2, then causing all the rest of the joins to be blocked indefinitely. It
    would also cause leader step down in a seemingly healthy cluster which is
    processing proposals. CheckQuorum was mandated by raft.ReadOnlyLeaseBased,
    which is a less safe option to do linearizable reads. Switch ReadOnlyOption
    back to raft.ReadOnlySafe. Moreover, we don't need to do quorum based lin
    reads in the Alpha servers, because of the switch to proposing and then
    applying transaction updates.
  • raft.ReadIndex is not working for some reason. So, commented out its usage in
    Zero (and removed it from Alpha permanently). Needs to be fixed when the
    following issue is resolved. etcd-io/etcd#9893
  • The logic to do lin reads was replicated in both Zero and Alpha. Refactor that
    into one place in conn/node.go.
  • Retry conf change proposals if they timeout. This mechanism is similar to the
    one introduced for normal proposals in a previous commit 06ea4c.
  • Use a lock to only allow one JoinCluster call at a time. Block JoinCluster
    until node.AddToCluster is successful (or return the error).
  • Set raft library to 3.2.23. Before upgrade, we were at 3.2.6.

This change is Reviewable

@manishrjain manishrjain merged commit eb3910c into master Jul 1, 2018
2 of 3 checks passed
@manishrjain manishrjain deleted the mrjn/joincluster branch Jul 1, 2018
dna2github pushed a commit to dna2fork/dgraph that referenced this issue Jul 19, 2019
This PR fixes dgraph-io#2286 .

- CheckQuorum was causing us multiple issues. When doing a 5-node Zero cluster
  bootstrap, it would cause a leader to step down when the size of the cluster
  is 2, then causing all the rest of the joins to be blocked indefinitely. It
  would also cause leader step down in a seemingly healthy cluster which is
  processing proposals. CheckQuorum was mandated by raft.ReadOnlyLeaseBased,
  which is a less safe option to do linearizable reads. Switch ReadOnlyOption
  back to raft.ReadOnlySafe. Moreover, we don't need to do quorum based lin
  reads in the Alpha servers, because of the switch to proposing and then
  applying transaction updates.
- raft.ReadIndex is not working for some reason. So, commented out its usage in
  Zero (and removed it from Alpha permanently). Needs to be fixed when the
  following issue is resolved. etcd-io/etcd#9893
- The logic to do lin reads was replicated in both Zero and Alpha. Refactor that
  into one place in conn/node.go.
- Retry conf change proposals if they timeout. This mechanism is similar to the
  one introduced for normal proposals in a previous commit 06ea4c.
- Use a lock to only allow one JoinCluster call at a time. Block JoinCluster
  until node.AddToCluster is successful (or return the error).
- Set raft library to 3.2.23. Before upgrade, we were at 3.2.6.

Commit log:

* Trying to understand why JoinCluster doesn't work properly.
* Fucking works. Fucking works.
* It all works now.
* More Dgraph servers. Found a new issue where requesting read quorum doesn't respond.
* Refactor wait lin read code and move it to conn/node.go
* Remove lin read wait for server, because txn timestamp should be sufficient for waiting. Also, for the time being, comment out lin read wait from Zero as well.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant