mig527: Velero debug tool

migration/migrating_3_4/troubleshooting-3-4.adoc

@@ -21,6 +21,8 @@ include::modules/migration-downloading-logs.adoc[leveloffset=+1]
 include::modules/migration-updating-deprecated-gvks.adoc[leveloffset=+1]
 include::modules/migration-error-messages.adoc[leveloffset=+1]
 include::modules/migration-dvm-error-node-selectors.adoc[leveloffset=+1]
+include::modules/migration-debugging-velero-resources.adoc[leveloffset=+1]
+include::modules/migration-partial-failure-velero.adoc[leveloffset=+2]
 include::modules/migration-using-must-gather.adoc[leveloffset=+1]
 
 [id="rolling-back-migration_{context}"]

migration/migrating_4_1_4/troubleshooting-4-1-4.adoc

@@ -20,6 +20,8 @@ include::modules/migration-using-mig-log-reader.adoc[leveloffset=+1]
 include::modules/migration-downloading-logs.adoc[leveloffset=+1]
 include::modules/migration-error-messages.adoc[leveloffset=+1]
 include::modules/migration-dvm-error-node-selectors.adoc[leveloffset=+1]
+include::modules/migration-debugging-velero-resources.adoc[leveloffset=+1]
+include::modules/migration-partial-failure-velero.adoc[leveloffset=+2]
 include::modules/migration-using-must-gather.adoc[leveloffset=+1]
 
 [id="rolling-back-migration_{context}"]

migration/migrating_4_2_4/troubleshooting-4-2-4.adoc

@@ -20,6 +20,8 @@ include::modules/migration-using-mig-log-reader.adoc[leveloffset=+1]
 include::modules/migration-downloading-logs.adoc[leveloffset=+1]
 include::modules/migration-error-messages.adoc[leveloffset=+1]
 include::modules/migration-dvm-error-node-selectors.adoc[leveloffset=+1]
+include::modules/migration-debugging-velero-resources.adoc[leveloffset=+1]
+include::modules/migration-partial-failure-velero.adoc[leveloffset=+2]
 include::modules/migration-using-must-gather.adoc[leveloffset=+1]
 
 [id="rolling-back-migration_{context}"]

modules/migration-debugging-velero-resources.adoc

@@ -0,0 +1,59 @@
+// Module included in the following assemblies:
+// * migration/migrating_3_4/troubleshooting-3-4.adoc
+// * migration/migrating_4_1_4/troubleshooting-4-1-4.adoc
+// * migration/migrating_4_2_4/troubleshooting-4-2-4.adoc
+
+[id="migration-debugging-velero-resources_{context}"]
+= Using the Velero CLI to debug Backup and Restore CRs
+
+You can debug the `Backup` and `Restore` custom resources (CRs) and partial migration failures with the Velero command line interface (CLI). The Velero CLI runs in the `velero` pod.
+
+[id="velero-command-syntax_{context}"]
+== Velero command syntax
+
+Velero CLI commands use the following syntax:
+[source,terminal]
+----
+$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero <resource> <command> <resource_id>
+----
+
+You can specify `velero-<pod> -n openshift-migration` in place of `$(oc get pods -n openshift-migration -o name | grep velero)`.
+
+[id="help-command_{context}"]
+== Help command
+
+The Velero `help` command lists all the Velero CLI commands:
+[source,terminal]
+----
+$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero --help
+----
+
+[id="describe-command_{context}"]
+== Describe command
+
+The Velero `describe` command provides a summary of warnings and errors associated with a Velero resource:
+[source,terminal]
+----
+$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero  <resource> describe <resource_id>
+----
+
+.Example
+[source,terminal]
+----
+$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero backup describe 0e44ae00-5dc3-11eb-9ca8-df7e5254778b-2d8ql
+----
+
+[id="logs-command_{context}"]
+== Logs command
+
+The Velero `logs` command provides the logs associated with a Velero resource:
+[source,terminal]
+----
+velero <resource> logs <resource_id>
+----
+
+.Example
+[source,terminal]
+----
+$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -- ./velero restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
+----

modules/migration-partial-failure-velero.adoc

@@ -0,0 +1,80 @@
+// Module included in the following assemblies:
+// * migration/migrating_3_4/troubleshooting-3-4.adoc
+// * migration/migrating_4_1_4/troubleshooting-4-1-4.adoc
+// * migration/migrating_4_2_4/troubleshooting-4-2-4.adoc
+
+[id="migration-partial-failure-velero_{context}"]
+= Debugging a partial migration failure
+
+You can debug a partial migration failure warning message by using the Velero CLI to examine the `Restore` custom resource (CR) logs.
+
+A partial failure occurs when Velero encounters an issue that does not cause a migration to fail. For example, if a custom resource definition (CRD) is missing or if there is a discrepancy between CRD versions on the source and target clusters, the migration completes but the CR is not created on the target cluster.
+
+Velero logs the issue as a partial failure and then processes the rest of the objects in the `Backup` CR.
+
+.Procedure
+
+. Check the status of a `MigMigration` CR:
++
+[source,terminal]
+----
+$ oc get migmigration <migmigration> -o yaml
+----
++
+.Example output
++
+[source,yaml]
+----
+status:
+  conditions:
+  - category: Warn
+    durable: true
+    lastTransitionTime: "2021-01-26T20:48:40Z"
+    message: 'Final Restore openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf: partially failed on destination cluster'
+    status: "True"
+    type: VeleroFinalRestorePartiallyFailed
+  - category: Advisory
+    durable: true
+    lastTransitionTime: "2021-01-26T20:48:42Z"
+    message: The migration has completed with warnings, please look at `Warn` conditions.
+    reason: Completed
+    status: "True"
+    type: SucceededWithWarnings
+----
+
+. Check the status of the `Restore` CR by using the Velero `describe` command:
++
+[source,yaml]
+----
+$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -n openshift-migration -- ./velero restore describe <restore>
+----
++
+.Example output
++
+[source,yaml]
+----
+Phase:  PartiallyFailed (run 'velero restore logs ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf' for more information)
+
+Errors:
+  Velero:     <none>
+  Cluster:    <none>
+  Namespaces:
+    migration-example:  error restoring example.com/migration-example/migration-example: the server could not find the requested resource
+----
+
+. Check the `Restore` CR logs by using the Velero `logs` command:
++
+[source,yaml]
+----
+$ oc exec $(oc get pods -n openshift-migration -o name | grep velero) -n openshift-migration -- ./velero restore logs <restore>
+----
++
+.Example output
++
+[source,yaml]
+----
+time="2021-01-26T20:48:37Z" level=info msg="Attempting to restore migration-example: migration-example" logSource="pkg/restore/restore.go:1107" restore=openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
+time="2021-01-26T20:48:37Z" level=info msg="error restoring migration-example: the server could not find the requested resource" logSource="pkg/restore/restore.go:1170" restore=openshift-migration/ccc7c2d0-6017-11eb-afab-85d0007f5a19-x4lbf
+----
++
+The `Restore` CR log error message, `the server could not find the requested resource`, indicates the cause of the partially failed migration.

modules/migration-rolling-back-migration-cli.adoc

@@ -6,20 +6,19 @@
 [id='migration-rolling-back-migration-cli_{context}']
 == Rolling back a migration from the CLI
 
-You can roll back a migration by using the CLI.
+You can roll back a migration by creating a `MigMigration` custom resource (CR) from the CLI.
 
 If your application was stopped during a failed migration, you must roll back the migration in order to prevent data corruption in the persistent volume.
 
 Rollback is not required if the application was not stopped during migration because the original application is still running on the source cluster.
 
 .Procedure
 
-. Create a `MigMigration` CR object based on the following example:
+. Create a `MigMigration` CR based on the following example:
 +
 [source,yaml]
 ----
 $ cat << EOF | oc apply -f -
----
 apiVersion: migration.openshift.io/v1alpha1
 kind: MigMigration
 metadata:
@@ -28,23 +27,15 @@ metadata:
   name: migration-rollback
   namespace: openshift-migration
 spec:
-  # 'canceled: true' cancels the migration
-  canceled: false
-  # 'rollback: true' rolls back the migration
+...
   rollback: true
-  # 'stage: true' runs a stage migration without quiescing the application on the source cluster.
-  stage: false
-  # 'quiescePods: true' scales the pods on the source cluster to '0' after the 'Backup' stage of a migration has finished
-  quiescePods: false
-  # 'keepAnnotations: true' retains the labels and annotations applied by the migration
-  keepAnnotations: false
-
+...
   migPlanRef:
-    name: <migplan-name> <1>
+    name: <migplan_name> <1>
     namespace: openshift-migration
 EOF
 ----
-<1> Specify the name of the migration plan that you want to roll back.
+<1> Specify the name of the associated `MigPlan` CR.
 
-. In the {mtc-short} console, verify that the migrated project resources have been removed from the target cluster.
+. In the {mtc-short} web console, verify that the migrated project resources have been removed from the target cluster.
 . Verify that the migrated project resources are present in the source cluster and that the application is running.