Further increase the default node-monitor-grace-period

[rtheis]

What happened?

Continuing the investigation started by #121793, we found that a 50 second default node-monitor-grace-period may not be sufficient since it doesn't account for a delta time needed by Kubernetes components to reconnect, coordinate, and complete their requests. For example, the kubelet sets node-status-update-frequency to 10 seconds by default which may only give the kubelet 4 retry attempts instead of the 5 desired.

What did you expect to happen?

All nodes stay Ready during a 40 second disconnection/keepalive timeout between the Kubernetes API server (KAS) and etcd resulting in the KAS failing over to another etcd member.

How can we reproduce it (as minimally and precisely as possible)?

See #121793 and use a 41 second etcd defrag time to account for a delta amount of time while the Kubernetes components re-establish network connections. While difficult to reproduce outside this simulation, we have had clients report a few problems that appear to confirm this situation may occur.

Anything else we need to know?

No

Kubernetes version

Details

Kubernetes version 1.29.8

Cloud provider

Details

IBM Cloud

OS version

Details

Ubuntu 20

Install tools

Details

IBM Cloud Kubernetes Service

Container runtime (CRI) and version (if applicable)

Details

containerd version 1.7.20

Related plugins (CNI, CSI, ...) and versions (if applicable)

Details

Calico version 3.27.4

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[rtheis]

Hypothetical Timeline: Unfortunately, we haven't been able to capture a detailed timeline yet to confirm this theory.

• 00:00:00.****: kubelet updates its lease.
• 00:00:09.9998: Kubernetes API server (KAS) completes etcd keepalived ping
• 00:00:09.9999: Problem starts
• 00:00:10.****: kubelet lease update attempt 1 fails
• 00:00:20.****: kubelet lease update attempt 2 fails
• 00:00:30.****: kubelet lease update attempt 3 fails
• 00:00:39.9998: KAS attempts etcd keepalived ping
• 00:00:40.****: kubelet lease update attempt 4 fails
• 00:00:49.9998: KAS etcd keepalived ping timeout, KAS needs to fix its etcd connection.
• 00:00:49.9999: Problem resolved
• 00:00:50.****: KAS connects to another etcd member.
• 00:00:50.****: Node lifecycle controller marks the node not ready.
• 00:00:50.****: kubelet updates its lease
• 00:00:52.0000: Node lifecycle controller marks the node ready again.

[rtheis]

We decided to set node-monitor-grace-period to 55 seconds to provide a very generous delta. This worked well in our testing.

[rtheis]

/sig node

[HirazawaUi]

/remove-kind bug
/kind feature

[bouaouda-achraf]

I can work on it .
/assign