Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster state does not always contain correct node roles #71385

Closed
DaveCTurner opened this issue Apr 7, 2021 · 2 comments
Closed

Cluster state does not always contain correct node roles #71385

DaveCTurner opened this issue Apr 7, 2021 · 2 comments
Labels
>bug :Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed Meta label for distributed team

Comments

@DaveCTurner
Copy link
Contributor

Elasticsearch version (bin/elasticsearch --version): 7.11.1, likely others

Plugins installed: []

JVM version (java -version): Bundled

OS version (uname -a if on a Unix-like system): CentOS Linux 7 (Core) 3.10.0-693.21.1.el7.x86_64

Description of the problem including expected versus actual behavior:

A cluster that was fully upgraded to a data-tiers-based 7.11.1 was seen to still have nodes with the data role in the cluster state:

$ cat nodes.json| jq '.nodes[].version' | uniq -c
  31 "7.11.1"
$ cat nodes.json| jq '.nodes[].settings.node.roles' -cMr | sort | uniq -c
  16 ["data_hot","data_content"]
   8 ["data_warm"]
   2 ["ingest","transform","remote_cluster_client"]
   3 ["master"]
   2 ["ml","ingest","transform","remote_cluster_client"]
$ cat cat/cat_nodes.txt | awk 'NR!=1 {print $5}' | sort | uniq -c
  12 d
   9 hs
   2 ilrt
   2 irt
   3 m
   3 w

There should be no d nodes in this cluster any more.

Steps to reproduce:

Unknown at the moment, this is exactly what #64693 is supposed to address.
@DaveCTurner DaveCTurner added >bug :Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. labels Apr 7, 2021
@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Apr 7, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 7, 2021
@DaveCTurner
Include node roles in cluster state JSON response
086852f 
Today the response to `GET _cluster/state` does not include the roles of
the nodes in the cluster. In the past this made sense, roles were
relatively unchanging things that could be determined from elsewhere.
These days we have an increasingly rich collection of roles, with
nontrivial BWC implications, so it is important for debugging to be able
to see the specific roles as viewed by the master. This commit adds the
role names to the cluster state API output.

Relates elastic#71385
@DaveCTurner DaveCTurner mentioned this issue Apr 7, 2021
Merged
DaveCTurner added a commit that referenced this issue Apr 7, 2021
@DaveCTurner
Include node roles in cluster state JSON response (#71386)
6ed2d25 
Today the response to `GET _cluster/state` does not include the roles of
the nodes in the cluster. In the past this made sense, roles were
relatively unchanging things that could be determined from elsewhere.
These days we have an increasingly rich collection of roles, with
nontrivial BWC implications, so it is important for debugging to be able
to see the specific roles as viewed by the master. This commit adds the
role names to the cluster state API output.

Relates #71385
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 7, 2021
@DaveCTurner
Include node roles in cluster state JSON response (elastic#71386)
6b42062 
Today the response to `GET _cluster/state` does not include the roles of
the nodes in the cluster. In the past this made sense, roles were
relatively unchanging things that could be determined from elsewhere.
These days we have an increasingly rich collection of roles, with
nontrivial BWC implications, so it is important for debugging to be able
to see the specific roles as viewed by the master. This commit adds the
role names to the cluster state API output.

Relates elastic#71385
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Apr 7, 2021
@DaveCTurner
Include node roles in cluster state JSON response
acf77d6 
Today the response to `GET _cluster/state` does not include the roles of
the nodes in the cluster. In the past this made sense, roles were
relatively unchanging things that could be determined from elsewhere.
These days we have an increasingly rich collection of roles, with
nontrivial BWC implications, so it is important for debugging to be able
to see the specific roles as viewed by the master. This commit adds the
role names to the cluster state API output.

Relates elastic#71385
Backport of elastic#71386
@DaveCTurner DaveCTurner mentioned this issue Apr 7, 2021
Merged
DaveCTurner added a commit that referenced this issue Apr 7, 2021
@DaveCTurner
Include node roles in cluster state JSON response (#71391)
6fb0cf6 
Today the response to `GET _cluster/state` does not include the roles of
the nodes in the cluster. In the past this made sense, roles were
relatively unchanging things that could be determined from elsewhere.
These days we have an increasingly rich collection of roles, with
nontrivial BWC implications, so it is important for debugging to be able
to see the specific roles as viewed by the master. This commit adds the
role names to the cluster state API output.

Relates #71385
Backport of #71386
@DaveCTurner DaveCTurner mentioned this issue Apr 16, 2021
Closed
@DaveCTurner
Copy link
Contributor Author

This only affects upgrades from versions that are now long-past EOL, so I'm closing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug :Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. Team:Distributed Meta label for distributed team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@DaveCTurner @elasticmachine