Skip to content

WIP OCPBUGS#9229: Graceful Shutdown improvements #45621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

WIP OCPBUGS#9229: Graceful Shutdown improvements #45621

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
203fc63
bz2076315: making bug revisions
nalhadef May 11, 2022
7007abb
Merge branch 'openshift:main' into bz2076315
nalhadef Sep 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 25 additions & 79 deletions modules/graceful-shutdown.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,110 +20,56 @@ You can shut down a cluster until a year from the installation date and expect i

.Procedure

. If you plan to shut down the cluster for an extended period of time, determine the date that cluster certificates expire.
+
You must restart the cluster prior to the date that certificates expire. As the cluster restarts, the process might require you to manually approve the pending certificate signing requests (CSRs) to recover kubelet certificates.

.. Check the expiration date for the `kube-apiserver-to-kubelet-signer` CA certificate:
. If you are shutting the cluster down for an extended period, determine the date on which certificates expire.
+
[source,terminal]
----
$ oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.metadata.annotations.auth\.openshift\.io/certificate-not-after}{"\n"}'
$ oc -n openshift-kube-apiserver-operator get secret kube-apiserver-to-kubelet-signer -o jsonpath='{.metadata.annotations.auth\.openshift\.io/certificate-not-after}'
----
+
.Example output
[source,terminal]
----
2023-08-05T14:37:50Z
----

.. Check the expiration date for the kubelet certificates:

... Start a debug session for a control plane node by running the following command:
+
[source,terminal]
----
$ oc debug node/<node_name>
----

... Change your root directory to `/host` by running the following command:
+
[source,terminal]
----
sh-4.4# chroot /host
2022-08-05T14:37:50Zuser@user:~ $ <1>
----
<1> To ensure that the cluster can restart gracefully, plan to restart it on or before the specified date. As the cluster restarts, the process might require you to manually approve the pending certificate signing requests (CSRs) to recover kubelet certificates.

... Check the kubelet client certificate expiration date by running the following command:
. Mark the control plane nodes in the cluster as unschedulable. You can do this from your cloud provider's web console, or run the following loop:
+
[source,terminal]
[source,yaml]
----
sh-5.1# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate
$ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm cordon ${node} ; done
----
+
.Example output
[source,terminal]
----
notAfter=Jun 6 10:50:07 2023 GMT
ci-ln-mgdnf4b-72292-n547t-master-0
node/ci-ln-mgdnf4b-72292-n547t-master-0 cordoned
ci-ln-mgdnf4b-72292-n547t-master-1
node/ci-ln-mgdnf4b-72292-n547t-master-1 cordoned
ci-ln-mgdnf4b-72292-n547t-master-2
node/ci-ln-mgdnf4b-72292-n547t-master-2 cordoned
ci-ln-mgdnf4b-72292-n547t-worker-a-s7ntl
node/ci-ln-mgdnf4b-72292-n547t-worker-a-s7ntl cordoned
ci-ln-mgdnf4b-72292-n547t-worker-b-cmc9k
node/ci-ln-mgdnf4b-72292-n547t-worker-b-cmc9k cordoned
ci-ln-mgdnf4b-72292-n547t-worker-c-vcmtn
node/ci-ln-mgdnf4b-72292-n547t-worker-c-vcmtn cordoned
----

... Check the kubelet server certificate expiration date by running the following command:
+
[source,terminal]
----
sh-5.1# openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -enddate
----
+
.Example output
[source,terminal]
----
notAfter=Jun 6 10:50:07 2023 GMT
----

... Exit the debug session.

... Repeat these steps to check certificate expiration dates on all control plane nodes. To ensure that the cluster can restart gracefully, plan to restart it before the earliest certificate expiration date.

. Shut down all of the nodes in the cluster. You can do this from your cloud provider's web console, or run the following loop:
+
[source,terminal]
----
$ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do oc debug node/${node} -- chroot /host shutdown -h 1; done <1>
----
<1> `-h 1` indicates how long, in minutes, this process lasts before the control-plane nodes are shut down. For large-scale clusters with 10 nodes or more, set to 10 minutes or longer to make sure all the compute nodes have time to shut down first.
. Evacuate the pods using the following method:
+
.Example output
[source,yaml]
----
Starting pod/ip-10-0-130-169us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Shutdown scheduled for Mon 2021-09-13 09:36:17 UTC, use 'shutdown -c' to cancel.

Removing debug pod ...
Starting pod/ip-10-0-150-116us-east-2computeinternal-debug ...
To use host binaries, run `chroot /host`
Shutdown scheduled for Mon 2021-09-13 09:36:29 UTC, use 'shutdown -c' to cancel.
$ for node in $(oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{.items[*].metadata.name}'); do echo ${node} ; oc adm drain ${node} --delete-emptydir-data --ignore-daemonsets=true --timeout=15s ; done
----
+
Shutting down the nodes using one of these methods allows pods to terminate gracefully, which reduces the chance for data corruption.
. Shut down all of the nodes in the cluster. You can do this from your cloud provider’s web console, or run the following loop:
+
[NOTE]
====
Adjust the shut down time to be longer for large-scale clusters:
[source,terminal]
[source,yaml]
----
$ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do oc debug node/${node} -- chroot /host shutdown -h 10; done
$ for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}'); do oc debug node/${node} -- chroot /host shutdown -h 1 ; done
----
====
+
[NOTE]
====
It is not necessary to drain control plane nodes of the standard pods that ship with {product-title} prior to shutdown.

Cluster administrators are responsible for ensuring a clean restart of their own workloads after the cluster is restarted. If you drained control plane nodes prior to shutdown because of custom workloads, you must mark the control plane nodes as schedulable before the cluster will be functional again after restart.
====

. Shut off any cluster dependencies that are no longer needed, such as external storage or an LDAP server. Be sure to consult your vendor's documentation before doing so.
+
[IMPORTANT]
====
If you deployed your cluster on a cloud-provider platform, do not shut down, suspend, or delete the associated cloud resources. If you delete the cloud resources of a suspended virtual machine, {product-title} might not restore successfully.
====