SNR fails to create peer list when `node-role.kubernetes.io/worker` label is missing

[hakuro95]

Description

Our team encountered an issue with SNR v0.9.0 in scenarios involving A Node without API-Server Access and An Isolated Node.

INFO    api-check       Error count exceeds threshold, trying to ask other nodes if I'm healthy
INFO    api-check       Peers list is empty and / or couldn't be retrieved from server, nothing we can do, so consider the node being healthy
INFO    api-check       peers did not confirm that we are unhealthy, ignoring error

SNR cannot query other nodes because the peer list is empty.

Analysis of the code reveals that SNR uses the node-role.kubernetes.io/worker label as a selector when creating the peer list.
Remediation functions as expected after setting this label.

Documenting this prerequisite would be beneficial.

Note: While the code responsible for the above log output has been modified in the latest version (Add minPeersForRemediation configuration value. · medik8s/self-node-remediation@194acec · GitHub), the underlying issue is likely still present.

Related files:

https://github.com/medik8s/self-node-remediation/blob/c960b00b7671d2b9f67134d18a73c08bde6be30c/pkg/apicheck/check.go#L132C1-L156C3

self-node-remediation/pkg/peers/peers.go

Line 11 in c960b00

commonlabels "github.com/medik8s/common/pkg/labels"

https://github.com/medik8s/self-node-remediation/blob/c960b00b7671d2b9f67134d18a73c08bde6be30c/pkg/peers/peers.go#L181C1-L182C37

self-node-remediation/pkg/peers/peers.go

Line 80 in c960b00

p.workerPeerSelector = createSelector(hostname, commonlabels.WorkerRole)

self-node-remediation/pkg/peers/peers.go

Line 86 in c960b00

updateWorkerPeersError := p.updateWorkerPeers(ctx)

https://github.com/medik8s/self-node-remediation/blob/c960b00b7671d2b9f67134d18a73c08bde6be30c/pkg/peers/peers.go#L104C1-L108C2

https://github.com/medik8s/common/blob/cd032bd2a08da3b2b8dded4fe066037ee5f1a94f/pkg/labels/labels.go#L5

[clobrano]

Hi @hakuro95 ,

thank you for submitting this problem. I'm having a look right now

[mshitrit]

@hakuro95 is there a particular use case you had in mind where node-role.kubernetes.io/worker would be missing ?

[hakuro95]

@mshitrit
I noticed that when I created a Kubernetes cluster using kubeadm or Kubespray, the node-role.kubernetes.io/worker label was not set.

[clobrano]

Hi @hakuro95, IIUC the role labels are indeed optional, we should probably document the need of such labels in SNR, thank you for reporting this problem.

[hakuro95]

Hi, @clobrano ,
I'd like to contribute to the medik8s SNR documentation based on issues #267 and #268. Specifically, I plan to explain the necessity of configuring node-role.kubernetes.io/worker and DNS resolution.
However, I'm not sure which section of the documentation would be most appropriate for this content.
I would appreciate it if the community could advise on the best place to include this information.

[clobrano]

That's awesome, thanks! Given this is the first time we've dealt with this, I think the "Troubleshooting" section is the right place for it. We could add a new subsection there to describe this specific problem.

[hakuro95]

Hi, @clobrano
I've already updated the medik8s SNR documentation based on issues #267 and #268.
Would you mind reviewing the PR below?
medik8s/docs#79

[hakuro95]

Hi, @clobrano
I've already addressed your review comments in the PR.
Please take a look when you have a moment.