Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start Master|Node fault detection pinging immediately during discovery #6706

Closed
wants to merge 1 commit into from

Conversation

bleskes
Copy link
Contributor

@bleskes bleskes commented Jul 3, 2014

After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  elastic#6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.
@martijnvg
Copy link
Member

LGTM

@bleskes bleskes closed this in ae16956 Jul 3, 2014
@bleskes bleskes removed the review label Jul 3, 2014
@bleskes bleskes deleted the immediately_ping branch July 3, 2014 12:53
bleskes added a commit that referenced this pull request Jul 3, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Jul 8, 2014
…ging"

In #6706 we change the master validation to start pining immediately after a new master as ellected or a node joined. The idea is to have a quicker response to failures. This does however create a problem if the new master has yet fully processed it's ellection and responds to the ping with a NoLongerMasterException. This causes the source node to remove the current master and ellect another, only to find out it's not a master either and so forth. We are moving this change to the feature/improve_zen branch, where the improvements we made will cause the situation to be handled properly.

This reverts commit ae16956.
bleskes added a commit that referenced this pull request Jul 8, 2014
…ging"

In #6706 we change the master validation to start pining immediately after a new master as ellected or a node joined. The idea is to have a quicker response to failures. This does however create a problem if the new master has yet fully processed it's ellection and responds to the ping with a NoLongerMasterException. This causes the source node to remove the current master and ellect another, only to find out it's not a master either and so forth. We are moving this change to the feature/improve_zen branch, where the improvements we made will cause the situation to be handled properly.

This reverts commit ae16956.
bleskes added a commit that referenced this pull request Jul 8, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Jul 9, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Jul 9, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Jul 9, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Jul 11, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Jul 11, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Jul 12, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Jul 12, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit to bleskes/elasticsearch that referenced this pull request Jul 14, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  elastic#6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes elastic#6706
bleskes added a commit to bleskes/elasticsearch that referenced this pull request Jul 14, 2014
This is to allow the master election to complete on the chosen master.

 Relates to elastic#6706
bleskes added a commit that referenced this pull request Jul 16, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Jul 16, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
@clintongormley clintongormley changed the title [Discovery] immediately start Master|Node fault detection pinging Resiliency: Start Master|Node fault detection pinging immediately during discovery Jul 16, 2014
bleskes added a commit that referenced this pull request Jul 17, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Jul 17, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Jul 22, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Jul 22, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Aug 4, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Aug 4, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Aug 5, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Aug 5, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Aug 14, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Aug 14, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Aug 22, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Aug 22, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Aug 23, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Aug 23, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit that referenced this pull request Aug 27, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Aug 27, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
bleskes added a commit to bleskes/elasticsearch that referenced this pull request Sep 1, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  elastic#6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes elastic#6706
bleskes added a commit to bleskes/elasticsearch that referenced this pull request Sep 1, 2014
This is to allow the master election to complete on the chosen master.

 Relates to elastic#6706
@bleskes bleskes added v1.4.0 and removed v1.3.0 labels Sep 2, 2014
bleskes added a commit that referenced this pull request Sep 8, 2014
After a node joins the clusters, it starts pinging the master to verify it's health. Before, the cluster join request was processed async and we had to give some time to complete. With  #6480 we changed this to wait for the join process to complete on the master. We can therefore start pinging immediately for fast detection of failures. Similar change can be made to the Node fault detection from the master side.

Closes #6706
bleskes added a commit that referenced this pull request Sep 8, 2014
This is to allow the master election to complete on the chosen master.

 Relates to #6706
@clintongormley clintongormley changed the title Resiliency: Start Master|Node fault detection pinging immediately during discovery Start Master|Node fault detection pinging immediately during discovery Jun 7, 2015
@clintongormley clintongormley added :Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. and removed :Cluster labels Feb 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Distributed A catch all label for anything in the Distributed Area. If you aren't sure, use this one. >enhancement resiliency v1.3.0 v2.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants