-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Updates rollback to SDN procedure to add live migration #81099
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
stevsmit
merged 1 commit into
openshift:enterprise-4.15
from
stevsmit:OSDOCS-11834-limited-live-migration-backport
Oct 14, 2024
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| // * networking/ovn_kubernetes_network_provider/migrate-from-openshift-sdn.adoc | ||
|
|
||
| ifeval::["{context}" == "migrate-to-openshift-sdn"] | ||
| :sdn: OpenShift SDN | ||
| :previous-sdn: OVN-Kubernetes | ||
| :type: OpenShiftSDN | ||
| endif::[] | ||
| ifeval::["{context}" == "migrate-from-openshift-sdn"] | ||
| :sdn: OVN-Kubernetes | ||
| :previous-sdn: OpenShift SDN | ||
| :type: OVNKubernetes | ||
| endif::[] | ||
|
|
||
| [id="how-the-live-migration-process-works_{context}"] | ||
| = How the limited live migration process works | ||
|
|
||
| The following table summarizes the limited live migration process by segmenting between the user-initiated steps in the process and the actions that the migration script performs in response. | ||
|
|
||
| .Limited live migration to OVNKubernetes from OpenShiftSDN | ||
| [cols="1,1a",options="header"] | ||
| |=== | ||
| |User-initiated steps|Migration activity | ||
| ifdef::openshift-rosa,openshift-dedicated[] | ||
| | Add the `unsupported-red-hat-internal-testing` annotation to the cluster-level network configuration. | ||
| | The Cluster Network Operator (CNO) acknowledges the unsupported testing environment. | ||
| endif::[] | ||
|
|
||
| | Patch the cluster-level networking configuration by changing the `networkType` from `OpenShiftSDN` to `OVNKubernetes`. | ||
| | | ||
| Cluster Network Operator (CNO):: | ||
| + | ||
| -- | ||
| * Sets migration related fields in the `network.operator` custom resource (CR) and waits for routable MTUs to be applied to all nodes. | ||
| * Patches the `network.operator` CR to set the migration mode to `Live` for OVN-Kubernetes and deploys the OpenShift SDN network plugin in migration mode. | ||
| * Deploys OVN-Kubernetes with hybrid overlay enabled, ensuring that no racing conditions occur. | ||
| * Waits for the OVN-Kubernetes deployment and updates the conditions in the status of the `network.config` CR. | ||
| * Triggers the Machine Config Operator (MCO) to apply the new machine config to each machine config pool, which includes node cordoning, draining, and rebooting. | ||
| * OVN-Kubernetes adds nodes to the appropriate zones and recreates pods using OVN-Kubernetes as the default CNI plugin. | ||
| * Removes migration-related fields from the network.operator CR and performs cleanup actions, such as deleting OpenShift SDN resources and redeploying OVN-Kubernetes in normal mode with the necessary configurations. | ||
| * Waits for the OVN-Kubernetes redeployment and updates the status conditions in the `network.config` CR to indicate migration completion. If your migration is blocked, see "Checking limited live migration metrics" for information on troubleshooting the issue. | ||
| -- | ||
| |=== | ||
|
|
||
| ifdef::sdn[] | ||
| :!sdn: | ||
| endif::[] | ||
| ifdef::previous-sdn[] | ||
| :!previous-sdn: | ||
| endif::[] | ||
| ifdef::type[] | ||
| :!type: | ||
| endif::[] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| // Module included in the following assemblies: | ||
| // | ||
| // * networking/ovn_kubernetes_network_provider/rollback-to-openshift-sdn.adoc | ||
|
|
||
| :_mod-docs-content-type: PROCEDURE | ||
| [id="nw-ovn-kubernetes-rollback-live_{context}"] | ||
| = Using the limited live migration method to roll back to the OpenShift SDN network plugin | ||
|
|
||
| As a cluster administrator, you can roll back to the OpenShift SDN Container Network Interface (CNI) network plugin by using the limited live migration method. During the migration with this method, nodes are automatically rebooted and service to the cluster is not interrupted. | ||
|
|
||
| [IMPORTANT] | ||
| ==== | ||
| You must wait until the migration process from OpenShift SDN to OVN-Kubernetes network plugin is successful before initiating a rollback. | ||
| ==== | ||
|
|
||
| If a rollback to OpenShift SDN is required, the following table describes the process. | ||
|
|
||
| .Performing a rollback to OpenShift SDN | ||
| [cols="1,1a",options="header"] | ||
| |=== | ||
|
|
||
| |User-initiated steps|Migration activity | ||
| ifdef::openshift-rosa,openshift-dedicated[] | ||
| | Add the `unsupported-red-hat-internal-testing` annotation to the cluster-level network configuration. | ||
| | The Cluster Network Operator (CNO) acknowledges the unsupported testing environment. | ||
| endif::[] | ||
|
|
||
| | Patch the cluster-level networking configuration by changing the `networkType` from `OVNKubernetes` to `OpenShiftSDN`. | ||
| | | ||
| Cluster Network Operator (CNO):: Performs the following actions: | ||
| + | ||
| -- | ||
| * Sets migration related fields in the `network.operator` custom resource (CR) and waits for routable MTUs to be applied to all nodes by the Machine Config Operator (MCO). | ||
| * Patches the `network.operator` CR to set the migration mode to `Live` for OpenShiftSDN and deploys the OpenShiftSDN network plugin in migration mode. | ||
| * Deploys OVN-Kubernetes with hybrid overlay enabled. | ||
| * Waits for both CNI plugins to be deployed and updates the conditions in the status of the `network.config` CR. | ||
| * Triggers the MCO to apply the new machine config to each machine config pool, which includes node cordoning, draining, and rebooting. | ||
| * Removes migration-related fields from the `network.operator` CR and performs cleanup actions, such as deleting OpenShift SDN resources and redeploying OVN-Kubernetes in normal mode with the necessary configurations. | ||
| * Waits for the OpenShiftSDN redeployment and updates the status conditions in the `network.config` CR to indicate migration completion. | ||
| -- | ||
|
|
||
| |=== | ||
|
|
||
| .Prerequisites | ||
|
|
||
| * The {oc-first} is installed. | ||
| * Access to the cluster as a user with the cluster-admin role is available. | ||
| * The cluster is installed on infrastructure configured with the OVN-Kubernetes network plugin. | ||
| * A recent backup of the etcd database is available. | ||
| * A manual reboot can be triggered for each node. | ||
| * The cluster is in a known good state, without any errors. | ||
| .Procedure | ||
|
|
||
| . To initiate the rollback to OpenShift SDN, enter the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ oc patch Network.config.openshift.io cluster --type='merge' --patch '{"metadata":{"annotations":{"network.openshift.io/network-type-migration":""}},"spec":{"networkType":"OpenShiftSDN"}}' | ||
| ---- | ||
|
|
||
| . To watch the progress of your migration, enter the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ watch -n1 'oc get network.config/cluster -o json | jq ".status.conditions[]|\"\\(.type) \\(.status) \\(.reason) \\(.message)\"" -r | column --table --table-columns NAME,STATUS,REASON,MESSAGE --table-columns-limit 4; echo; oc get mcp -o wide; echo; oc get node -o "custom-columns=NAME:metadata.name,STATE:metadata.annotations.machineconfiguration\\.openshift\\.io/state,DESIRED:metadata.annotations.machineconfiguration\\.openshift\\.io/desiredConfig,CURRENT:metadata.annotations.machineconfiguration\\.openshift\\.io/currentConfig,REASON:metadata.annotations.machineconfiguration\\.openshift\\.io/reason"' | ||
| ---- | ||
| + | ||
| The command prints the following information every second: | ||
| + | ||
| * The conditions on the status of the `network.config.openshift.io/cluster` object, reporting the progress of the migration. | ||
| * The status of different nodes with respect to the `machine-config-operator` resource, including whether they are upgrading or have been upgraded, as well as their current and desired configurations. | ||
|
|
||
| . Complete the following steps only if the migration succeeds and your cluster is in a good state: | ||
|
|
||
| .. Remove the `network.openshift.io/network-type-migration=` annotation from the `network.config` custom resource by entering the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ oc annotate network.config cluster network.openshift.io/network-type-migration- | ||
| ---- | ||
|
|
||
| .. Remove the OVN-Kubernetes network provider namespace by entering the following command: | ||
| + | ||
| [source,terminal] | ||
| ---- | ||
| $ oc delete namespace openshift-ovn-kubernetes | ||
| ---- | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -6,24 +6,60 @@ | |||||||||||||||||||||||||
| // This procedure applies to both a roll back and a migration | ||||||||||||||||||||||||||
| :_mod-docs-content-type: PROCEDURE | ||||||||||||||||||||||||||
| [id="nw-ovn-kubernetes-rollback_{context}"] | ||||||||||||||||||||||||||
| = Migrating to the OpenShift SDN network plugin | ||||||||||||||||||||||||||
| = Using the offline migration method to roll back to the OpenShift SDN network plugin | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| Cluster administrators can roll back to the OpenShift SDN Container Network Interface (CNI) network plugin by using the offline migration method. During the migration you must manually reboot every node in your cluster. With the offline migration method, there is some downtime, during which your cluster is unreachable. | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| ifeval::["{context}" == "rollback-to-openshift-sdn"] | ||||||||||||||||||||||||||
| [IMPORTANT] | ||||||||||||||||||||||||||
| ==== | ||||||||||||||||||||||||||
| You must wait until the migration process from OpenShift SDN to OVN-Kubernetes network plugin is successful before initiating a rollback. | ||||||||||||||||||||||||||
| ==== | ||||||||||||||||||||||||||
| endif::[] | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| If a rollback to OpenShift SDN is required, the following table describes the process. | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| .Performing a rollback to OpenShift SDN | ||||||||||||||||||||||||||
| [cols="1,1a",options="header"] | ||||||||||||||||||||||||||
| |=== | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| |User-initiated steps|Migration activity | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| |Suspend the MCO to ensure that it does not interrupt the migration. | ||||||||||||||||||||||||||
| |The MCO stops. | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| | | ||||||||||||||||||||||||||
| Set the `migration` field of the `Network.operator.openshift.io` custom resource (CR) named `cluster` to `OpenShiftSDN`. Make sure the `migration` field is `null` before setting it to a value. | ||||||||||||||||||||||||||
| | | ||||||||||||||||||||||||||
| CNO:: Updates the status of the `Network.config.openshift.io` CR named `cluster` accordingly. | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| |Update the `networkType` field. | ||||||||||||||||||||||||||
| | | ||||||||||||||||||||||||||
| CNO:: Performs the following actions: | ||||||||||||||||||||||||||
| + | ||||||||||||||||||||||||||
| -- | ||||||||||||||||||||||||||
| * Destroys the OVN-Kubernetes control plane pods. | ||||||||||||||||||||||||||
| * Deploys the OpenShift SDN control plane pods. | ||||||||||||||||||||||||||
| * Updates the Multus objects to reflect the new network plugin. | ||||||||||||||||||||||||||
| -- | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| | | ||||||||||||||||||||||||||
| Reboot each node in the cluster. | ||||||||||||||||||||||||||
| | | ||||||||||||||||||||||||||
| Cluster:: As nodes reboot, the cluster assigns IP addresses to pods on the OpenShift-SDN network. | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| | | ||||||||||||||||||||||||||
| Enable the MCO after all nodes in the cluster reboot. | ||||||||||||||||||||||||||
| | | ||||||||||||||||||||||||||
| MCO:: Rolls out an update to the systemd configuration necessary for OpenShift SDN; the MCO updates a single machine per pool at a time by default, so the total time the migration takes increases with the size of the cluster. | ||||||||||||||||||||||||||
|
Comment on lines
+47
to
+52
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see "OpenShift-SDN" and "OpenShift SDN":
Suggested change
|
||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| |=== | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| .Prerequisites | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| * Install the OpenShift CLI (`oc`). | ||||||||||||||||||||||||||
| * Access to the cluster as a user with the `cluster-admin` role. | ||||||||||||||||||||||||||
| * A cluster installed on infrastructure configured with the OVN-Kubernetes network plugin. | ||||||||||||||||||||||||||
| * The {oc-first} is installed. | ||||||||||||||||||||||||||
| * Access to the cluster as a user with the cluster-admin role is available. | ||||||||||||||||||||||||||
| * The cluster is installed on infrastructure configured with the OVN-Kubernetes network plugin. | ||||||||||||||||||||||||||
| * A recent backup of the etcd database is available. | ||||||||||||||||||||||||||
| * A reboot can be triggered manually for each node. | ||||||||||||||||||||||||||
| * A manual reboot can be triggered for each node. | ||||||||||||||||||||||||||
| * The cluster is in a known good state, without any errors. | ||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||
| .Procedure | ||||||||||||||||||||||||||
|
|
@@ -357,4 +393,4 @@ $ oc patch Network.operator.openshift.io cluster --type='merge' \ | |||||||||||||||||||||||||
| [source,terminal] | ||||||||||||||||||||||||||
| ---- | ||||||||||||||||||||||||||
| $ oc delete namespace openshift-ovn-kubernetes | ||||||||||||||||||||||||||
| ---- | ||||||||||||||||||||||||||
| ---- | ||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mark these up? Or switch to "OpenShift SDN" if a general reference?