CRD Upgrade Check fails with lots of "unknown change, refusing to determine that change is safe"

[joelanford]

I installed argocd-operator 0.6.0 successfully, and then tried an upgrade to 0.7.0. I expected the upgrade to succeed, but I got the following error message in the ClusterExtension's Progressing condition.

validating upgrade for CRD "applicationsets.argoproj.io" failed: CustomResourceDefinition applicationsets.argoproj.io failed upgrade safety validation. "ChangeValidator" validation failed: version "v1alpha1", field "^.spec.generators" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].clusterDecisionResource.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].clusters.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].git.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].list.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].matrix.generators" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].matrix.generators[*].clusterDecisionResource.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].matrix.generators[*].clusters.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].matrix.generators[*].git.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].matrix.generators[*].list.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].matrix.generators[*].pullRequest.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].matrix.generators[*].scmProvider.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].matrix.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].merge.generators" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].merge.generators[*].clusterDecisionResource.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].merge.generators[*].clusters.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].merge.generators[*].git.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].merge.generators[*].list.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].merge.generators[*].pullRequest.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].merge.generators[*].scmProvider.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].merge.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].pullRequest.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.generators[*].scmProvider.template.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.template.spec.sources" has unknown change, refusing to determine that change is safe
validating upgrade for CRD "applications.argoproj.io" failed: CustomResourceDefinition applications.argoproj.io failed upgrade safety validation. "ChangeValidator" validation failed: version "v1alpha1", field "^.operation.sync.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.spec.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.status.history" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.status.history[*].sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.status.operationState.operation.sync.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.status.operationState.syncResult.sources" has unknown change, refusing to determine that change is safe
version "v1alpha1", field "^.status.sync.comparedTo.sources" has unknown change, refusing to determine that change is safe

[joelanford]

It looks like maybe the CalculateFlatSchemaDiff function in carvel.dev/kapp/pkg/kapp/crdupgradesafety/change_validator.go may need to set oldCopy.Items and newCopy.Items to nil as well, which would align with the intent of the comment just above the DeepCopy calls.

https://github.com/carvel-dev/kapp/blob/d7fc2e15439331aa3a379485bb124e91a0829d2e/pkg/kapp/crdupgradesafety/change_validator.go#L543-L544

By setting Items to nil, I can confirm that the upgrade proceeds successfully.

[azych]

potential duplicate with #1854