Bug 1999556: annotate rendered config with OCP version

[kikisdeliveryservice]

We need to ensure that all rendered configs were generated in the proper version of the OCP release version before allowing the MCO to report upgraded. Previously, we added checks to verify MCs were generated with the correct MCO commit/version.Hash and verify the osImageURL. However, on rare occasions (and usually just in ci) there will be an OCP release that neither includes a new MCO commit nor a new rhcos image, and because we aren't checking against the OCP version, we prematurely report upgraded before the new MCO/MCC/Rendered MCs are generated.

• added annotation of release version to rendered configs and controller config :

         "metadata": {
            "annotations": {
                "machineconfiguration.openshift.io/release-image-version": "4.10.0-0.ci.test-2022-01-20-003840-ci-op-mnyl122f-latest"
            }

[openshift-ci]

@kikisdeliveryservice: This pull request references Bugzilla bug 1999556, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.10.0) matches configured target release for branch (4.10.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (rioliu@redhat.com), skipping review request.

In response to this:

[WIP] Bug 1999556: [ignore]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@kikisdeliveryservice: This pull request references Bugzilla bug 1999556, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.10.0) matches configured target release for branch (4.10.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (rioliu@redhat.com), skipping review request.

In response to this:

[WIP] Bug 1999556: [ignore]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kikisdeliveryservice]

/test e2e-aws-disruptive

[openshift-ci]

@kikisdeliveryservice: This pull request references Bugzilla bug 1999556, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.10.0) matches configured target release for branch (4.10.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (rioliu@redhat.com), skipping review request.

In response to this:

[WIP] Bug 1999556: [ignore]

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@kikisdeliveryservice: This pull request references Bugzilla bug 1999556, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.10.0) matches configured target release for branch (4.10.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (rioliu@redhat.com), skipping review request.

In response to this:

[WIP] Bug 1999556: annotate rendered config with OCP version

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kikisdeliveryservice]

mco/pr looking good, but some unrelatedblocked passing

``multus-validating-config.k8s.io: x509: certificate signed by unknown authority"```

/test e2e-agnostic-upgrade

[kikisdeliveryservice]

aws not doing great
/test e2e-aws

[kikisdeliveryservice]

error: error creating buildah builder: Error reading blob ...invalid status code from registry 503 (Service Unavailable) on both retests =/

/retest-required

[cgwalters]

Overall, this looks sane to me and I think we could ship it as is. Some comments below which could be considered all optionals/informational.

[cgwalters]

Previous related PRs:

• dd54b84
• 60c5ee2
• db4ca30

[kikisdeliveryservice]

A couple of changes are incoming

/hold

[kikisdeliveryservice]

Updated PR after doing some testing to implement some of @cgwalters suggestions. This is now ready. 😸

/hold cancel

[yuqi-zhang]

Code generally lgtm. I have a question on the BZ: for situations that would trigger the bugzilla error
the "master" pool should be updated before the CVO reports available at the new version occurred

The master pool would have to be marked as Updating when the MCO operator reports updated right? In the case where neither the MCO commit changes nor the OSImage changes, wouldn't the master pool not roll out any update? i.e. the pool never goes to updating and we wouldn't even see the error in the bugzilla?

Looking through CI search (https://search.ci.openshift.org/?search=pool+should+be+updated+before+the+CVO+reports+available+at+the+new+version&maxAge=336h&context=1&type=junit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job), there was only 2 hits of this in the past 2 weeks. Looking at the first link, there was a new rendered-master being generated:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.7-e2e-metal-ipi-upgrade/1483817445192896512/artifacts/e2e-metal-ipi-upgrade/gather-extra/artifacts/oc_cmds/machineconfigs

Comparing those 2 in https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.7-e2e-metal-ipi-upgrade/1483817445192896512/artifacts/e2e-metal-ipi-upgrade/gather-extra/artifacts/machineconfigs.json

it seems that the upgrade only contained a diff in
/etc/kubernetes/manifests/keepalived.yaml

Which seems to imply that to trigger this error, a template change would be the only diff between the two OCP release versions...? Or is that due to another clusteroperator change that changed the dynamic variables populating that file?

Would this PR be covering this scenario? Sorry, am a bit confused as to how we can trigger this error after your previous fixes, so trying to think this through.

[kikisdeliveryservice]

@yuqi-zhang Intention for this PR is to fix the known issue of when there is an OCP upgrade that contains no new MCO commit and no new MCO osImageURL. Have had known failures there in the past, so closing the loop on that case via this pr by waiting for a new renderconfig to always roll out on all upgrades.

In the run that you pointed to, osimageurl is same for both releases and old rendered config and new rendered config both point to same controller version 51dc0801ed7d705820f557fcabf04eff023bf568. So, this PR should cover that case (ofc ymmv since that cluster hit many problems and didn't actually upgrade) :)

[yuqi-zhang]

Yeah ok that makes sense, so we don't report before the newest render gets generated.

/lgtm

To give it a safer chance of making it in

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cgwalters, kikisdeliveryservice, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [cgwalters,kikisdeliveryservice,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-bot]

/retest-required

Please review the full test history for this PR and help us cut down flakes.

[openshift-ci]

@kikisdeliveryservice: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-single-node	c5b9bf0	link	false	/test e2e-aws-single-node
ci/prow/e2e-aws-disruptive	c5b9bf0	link	false	/test e2e-aws-disruptive
ci/prow/e2e-aws-upgrade-single-node	c5b9bf0	link	false	/test e2e-aws-upgrade-single-node
ci/prow/e2e-gcp-op-single-node	c5b9bf0	link	false	/test e2e-gcp-op-single-node
ci/prow/e2e-aws-serial	c5b9bf0	link	false	/test e2e-aws-serial
ci/prow/e2e-aws-workers-rhel7	c5b9bf0	link	false	/test e2e-aws-workers-rhel7
ci/prow/e2e-vsphere-upgrade	c5b9bf0	link	false	/test e2e-vsphere-upgrade
ci/prow/okd-e2e-aws	c5b9bf0	link	false	/test okd-e2e-aws
ci/prow/e2e-aws-workers-rhel8	c5b9bf0	link	false	/test e2e-aws-workers-rhel8

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-ci]

@kikisdeliveryservice: All pull requests linked via external trackers have merged:

• openshift/machine-config-operator#2918

Bugzilla bug 1999556 has been moved to the MODIFIED state.

In response to this:

Bug 1999556: annotate rendered config with OCP version

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kikisdeliveryservice]

/cherry-pick release-4.9

[openshift-cherrypick-robot]

@kikisdeliveryservice: #2918 failed to apply on top of branch "release-4.9":

Applying: mco/mcc: annotate rendered config with OCP version
Using index info to reconstruct a base tree...
M	pkg/controller/common/constants.go
M	pkg/controller/render/render_controller.go
M	pkg/operator/status.go
M	pkg/operator/sync.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/operator/sync.go
CONFLICT (content): Merge conflict in pkg/operator/sync.go
Auto-merging pkg/operator/status.go
Auto-merging pkg/controller/render/render_controller.go
Auto-merging pkg/controller/common/constants.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 mco/mcc: annotate rendered config with OCP version
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-4.9

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.