OCPBUGS-33275: extramanifest configmaps validation update

[Missxiaoguo]

Background / Context

LCA performs Kubernetes Dry-run to validate the manifests included in the extraManifests configmaps. Any invalid/webhook badRequest type of error returned from dry-run fails the Prep stage and blocks the upgrade.

Issue / Requirement / Reason for change

Kubernetes Dry-run sends requests to API server, where the server processes the request as if it were real, but without acutally making any modifications to the actual resources. As all API requests are single-resource requests, dryrun doesn't work for resources that depends on eariler resources that are not persisted in the cluster. For example, in a multi-object manifest that includes namespace A and resource A created within the namespace A. Running a dry-run on resource returns the namespace not found error.

The extra manifests are configurations intented to be applied to the new cluster version. Runing dry-run on the old version aims to thoroughly check for possible manifest error before upgrading. However, in cases where extramanifests configmaps contain the dependent resources, the Dry-run failure results in failed Prep, but the failure may be not the case for the new cluster version.

Solution / Feature Overview

Not only in the case of dependent resources, but there may also be other dry-run errors that occur only on the old cluster version. The dry-run validation should be a soft validation. The errors such as invalid or webhook badRequest type of error from dry-run are considered warnings. This also includes cases where CRDs are missing from the old version and dependent namespace does not exist on the old version but is also not found in the configmaps. The warnings do not block upgrade. Instead, the Prep status condition could be updated with a warning message and the failure details can be provided in an IBU annotation. The other type of errors such as random chars, missing resource Kind, resource ApiVersion, resource name or the presence of disallowed resource types, are treated as critical errors which will fail the Prep.

Implementation Details

Other Information

• If user uses the ZTP/TALM to run IBU with extra manifests configmaps, the prep stage policy should include message check with "Prep stage completed successfully" which would prevent CGU to complete if there's validation warning

• Log error in the upgrade handler for the error that causes the upgrade to fail.

/cc @donpenney @pixelsoccupied @leo8a

[openshift-ci-robot]

@Missxiaoguo: This pull request references Jira Issue OCPBUGS-33275, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (trwest@redhat.com), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Background / Context

LCA performs Kubernetes Dry-run to validate the manifests included in the extraManifests configmaps. Any invalid/webhook badRequest type of error returned from dry-run fails the Prep stage and blocks the upgrade.

Issue / Requirement / Reason for change

Kubernetes Dry-run sends requests to API server, where the server processes the request as if it were real, but without acutally making any modifications to the actual resources. As all API requests are single-resource requests, dryrun doesn't work for resources that depends on eariler resources that are not persisted in the cluster. For example, in a multi-object manifest that includes namespace A and resource A created within the namespace A. Running a dry-run on resource returns the namespace not found error.

The extra manifests are configurations intented to be applied to the new cluster version. Runing dry-run on the old version aims to thoroughly check for possible manifest error before upgrading. However, in cases where extramanifests configmaps contain the dependent resources, the Dry-run failure results in failed Prep, but the failure may be not the case for the new cluster version.

Solution / Feature Overview

Not only in the case of dependent resources, but there may also be other dry-run errors that occur only on the old cluster version. The dry-run validation should be a soft validation. The errors such as invalid or webhook badRequest type of error from dry-run are considered warnings. This also includes cases where CRDs are missing from the old version and dependent namespace does not exist on the old version but is also not found in the configmaps. The warnings do not block upgrade. Instead, the Prep status condition could be updated with a warning message and the failure details can be provided in an IBU annotation. The other type of errors such as random chars, missing resource Kind, resource ApiVersion, resource name or the presence of disallowed resource types, are treated as critical errors which will fail the Prep.

Implementation Details

Other Information

• If user uses the ZTP/TALM to run IBU with extra manifests configmaps, the prep stage policy should include message check with "Prep stage completed successfully" which would prevent CGU to complete if there's validation warning

• Log error in the upgrade handler for the error that causes the upgrade to fail.

/cc @donpenney @pixelsoccupied @leo8a

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[leo8a]

nice work Angie, thanks

dropped some minor comments below, mainly related to code style guidelines

[jc-rh]

/cc @sudomakeinstall2

If user uses the ZTP/TALM to run IBU with extra manifests configmaps, the prep stage policy should include message check with "Prep stage completed successfully" which would prevent CGU to complete if there's validation warning

[pixelsoccupied]

Just tiny comments! But lgtm otherwise

[Missxiaoguo]

/retest-required

[leo8a]

/lgtm

[pixelsoccupied]

/lgtm

[pixelsoccupied]

/lgtm

[donpenney]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: donpenney

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [donpenney]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[pixelsoccupied]

/test ibu-e2e-flow

[donpenney]

/override ci/prow/integration ci/prow/ibu-e2e-flow

[openshift-ci]

@donpenney: Overrode contexts on behalf of donpenney: ci/prow/ibu-e2e-flow, ci/prow/integration

In response to this:

/override ci/prow/integration ci/prow/ibu-e2e-flow

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[donpenney]

/override ci/prow/ci-index

[openshift-ci]

@donpenney: Overrode contexts on behalf of donpenney: ci/prow/ci-index

In response to this:

/override ci/prow/ci-index

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci-robot]

@Missxiaoguo: Jira Issue OCPBUGS-33275: All pull requests linked via external trackers have merged:

• openshift-kni/lifecycle-agent#521

Jira Issue OCPBUGS-33275 has been moved to the MODIFIED state.

In response to this:

Background / Context

LCA performs Kubernetes Dry-run to validate the manifests included in the extraManifests configmaps. Any invalid/webhook badRequest type of error returned from dry-run fails the Prep stage and blocks the upgrade.

Issue / Requirement / Reason for change

Kubernetes Dry-run sends requests to API server, where the server processes the request as if it were real, but without acutally making any modifications to the actual resources. As all API requests are single-resource requests, dryrun doesn't work for resources that depends on eariler resources that are not persisted in the cluster. For example, in a multi-object manifest that includes namespace A and resource A created within the namespace A. Running a dry-run on resource returns the namespace not found error.

The extra manifests are configurations intented to be applied to the new cluster version. Runing dry-run on the old version aims to thoroughly check for possible manifest error before upgrading. However, in cases where extramanifests configmaps contain the dependent resources, the Dry-run failure results in failed Prep, but the failure may be not the case for the new cluster version.

Solution / Feature Overview

Not only in the case of dependent resources, but there may also be other dry-run errors that occur only on the old cluster version. The dry-run validation should be a soft validation. The errors such as invalid or webhook badRequest type of error from dry-run are considered warnings. This also includes cases where CRDs are missing from the old version and dependent namespace does not exist on the old version but is also not found in the configmaps. The warnings do not block upgrade. Instead, the Prep status condition could be updated with a warning message and the failure details can be provided in an IBU annotation. The other type of errors such as random chars, missing resource Kind, resource ApiVersion, resource name or the presence of disallowed resource types, are treated as critical errors which will fail the Prep.

Implementation Details

Other Information

• If user uses the ZTP/TALM to run IBU with extra manifests configmaps, the prep stage policy should include message check with "Prep stage completed successfully" which would prevent CGU to complete if there's validation warning

• Log error in the upgrade handler for the error that causes the upgrade to fail.

/cc @donpenney @pixelsoccupied @leo8a

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.