Let payload provider render feature gate yaml #2664

enxebre · 2023-06-08T16:47:04Z

Currently we render the feature gate at the HCP level which results in a discrepancy when mco consume it along with a given NodePool payload-version that differs from the one used by the HCP to generate it. Related #2643

CI Failure example
bootstrap.go:47] error running MCC[BOOTSTRAP]: error creating feature gate access: unable to determine features: missing desired version "4.14.0-0.ci.test-2023-06-08-101141-ci-op-gvin17ww-initial" in featuregates.config.openshift.io/cluster\n"}

https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/3724/pull-ci-openshift-machine-config-operator-master-e2e-hypershift/1666749702499995648

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/3724/pull-ci-openshift-machine-config-operator-master-e2e-hypershift/1666749702499995648/artifacts/e2e-hypershift/run-e2e/artifacts/TestNodePool_PreTeardownClusterDump/namespaces/e2e-clusters-wvxmd-example-fmxk8/core/pods/logs/ignition-server-768699fd7b-w49lq-ignition-server.log

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #

Checklist

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

csrwng

Just one comment, but if it fixes the current issue, we can definitely follow up.

csrwng · 2023-06-08T17:44:26Z

control-plane-operator/controllers/hostedcontrolplane/ignitionserver/ignitionserver.go

@@ -437,7 +437,7 @@ func reconcileDeployment(deployment *appsv1.Deployment,
 						},
 						Args: []string{
 							"-c",
-							invokeFeatureGateRenderScript("/shared", releaseVersion, string(featureGateYAML)),
+							invokeFeatureGateRenderScript("/shared", string(featureGateYAML)),


If all this container is doing is writing out the feature gate yaml, we can likely do without it. The local ignition provider could read the hcp directly and create the yaml.

Makes sense, I'm wanting to keep this PR scope to the minimum.
In follow up we can also consider to let the NodePool controller store the featureGate yaml into the token secret so then we just pass it through the getPayload signature.

openshift-ci · 2023-06-08T17:47:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csrwng, enxebre

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [csrwng,enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

csrwng · 2023-06-08T18:14:12Z

/hold

enxebre · 2023-06-09T08:58:11Z

cluster config operator script failed with

{"level":"info","ts":"2023-06-08T17:11:00Z","logger":"get-payload","msg":"cluster-config-operator process completed","time":"0s","output":"/bin/bash: line 19: /usr/bin/cluster-config-operator: No such file or directory\n"}

and

{"level":"info","ts":"2023-06-08T17:11:32Z","logger":"get-payload","msg":"cluster-config-operator process completed","time":"0s","output":"mkdir: cannot create directory 'input': File exists\nmkdir: cannot create directory 'output': File exists\nmkdir: cannot create directory 'manifests': File exists\n"}

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2664/pull-ci-openshift-hypershift-main-e2e-aws/1666849350090756096/artifacts/e2e-aws/run-e2e/artifacts/TestCreateCluster_PreTeardownClusterDump/namespaces/e2e-clusters-gtp7b-example-jqvp2/core/pods/logs/ignition-server-7b587d498c-vslgl-ignition-server.log
updated

enxebre · 2023-06-09T12:37:07Z

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2664/pull-ci-openshift-hypershift-main-e2e-aws/1667120840149307392/artifacts/e2e-aws/run-e2e/artifacts/TestNodePool_PreTeardownClusterDump/namespaces/e2e-clusters-55smd-example-498fs/core/pods/logs/ignition-server-7b9886b8b7-cf9d6-ignition-server.log

{"level":"info","ts":"2023-06-09T11:10:02Z","logger":"get-payload","msg":"cluster-config-operator process completed","time":"0s","output":"F0609 11:10:02.959705      33 render.go:53] failed rendering assets: lstat /usr/share/bootkube/manifests/bootstrap-manifests: no such file or directory\n"}

netlify · 2023-06-13T08:24:15Z

✅ Deploy Preview for hypershift-docs ready!

Name	Link
🔨 Latest commit	`8093e27`
🔍 Latest deploy log	https://app.netlify.com/sites/hypershift-docs/deploys/64888c3454879e0008f603c0
😎 Deploy Preview	https://deploy-preview-2664--hypershift-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

enxebre · 2023-06-13T10:33:35Z

we are back to only upgrade failing, now with
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2664/pull-ci-openshift-hypershift-main-e2e-aws/1668534680921575424/artifacts/e2e-aws/run-e2e/artifacts/TestNodePool_PreTeardownClusterDump/namespaces/e2e-clusters-kzpgc-example-sqbgk/core/pods/logs/ignition-server-567875cf5b-z8rws-ignition-server.log

{"level":"info","ts":"2023-06-13T08:52:01Z","logger":"get-payload","msg":"cluster-config-operator process completed","time":"0s","output":"Error: unknown flag: --rendered-manifest-files\n"}

JoelSpeed · 2023-06-13T11:06:24Z

we are back to only upgrade failing, now with https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_hypershift/2664/pull-ci-openshift-hypershift-main-e2e-aws/1668534680921575424/artifacts/e2e-aws/run-e2e/artifacts/TestNodePool_PreTeardownClusterDump/namespaces/e2e-clusters-kzpgc-example-sqbgk/core/pods/logs/ignition-server-567875cf5b-z8rws-ignition-server.log
{"level":"info","ts":"2023-06-13T08:52:01Z","logger":"get-payload","msg":"cluster-config-operator process completed","time":"0s","output":"Error: unknown flag: --rendered-manifest-files\n"}

This looks like a mismatch between the payload version and the flags, is the test upgrading from a 4.14 to 4.14 or from 4.13 to 4.14? If it is upgrading from 4.13, we need to make sure it only passes the --rendered-manifest-files flag to the 4.14 payload not the 4.13 payload

muraee · 2023-06-13T14:46:05Z

ignition-server/controllers/local_ignitionprovider.go

+%[1]s render \
+   --config-output-file config \
+   --asset-input-dir %[2]s/input \
+   --asset-output-dir %[2]s/output


don't we need to pass --payload-version here ?

no, only for >= 4.14

muraee · 2023-06-13T15:08:27Z

lgtm

enxebre · 2023-06-13T15:09:14Z

~~/test e2e-aws-4-12~~

Ignore, this should make no difference as this cpo is not running there.

csrwng · 2023-06-13T15:29:09Z

/lgtm

enxebre · 2023-06-13T15:32:21Z

/hold cancel

Currently we render the feature gate at the HCP level which results in a discrepancy when mco consume it along with a given NodePool payload-version that differs from the one used by the HCP to generate it. Related openshift#2643 CI Failure example bootstrap.go:47] error running MCC[BOOTSTRAP]: error creating feature gate access: unable to determine features: missing desired version \"4.14.0-0.ci.test-2023-06-08-101141-ci-op-gvin17ww-initial\" in featuregates.config.openshift.io/cluster\n"} https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/3724/pull-ci-openshift-machine-config-operator-master-e2e-hypershift/1666749702499995648 https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_machine-config-operator/3724/pull-ci-openshift-machine-config-operator-master-e2e-hypershift/1666749702499995648/artifacts/e2e-hypershift/run-e2e/artifacts/TestNodePool_PreTeardownClusterDump/namespaces/e2e-clusters-wvxmd-example-fmxk8/core/pods/logs/ignition-server-768699fd7b-w49lq-ignition-server.log

openshift-ci · 2023-06-13T15:33:11Z

New changes are detected. LGTM label has been removed.

enxebre · 2023-06-13T15:33:27Z

rebased to pick Red Hat Trusted App Pipeline CI fix
/lgtm

openshift-ci · 2023-06-13T15:33:32Z

@enxebre: you cannot LGTM your own PR.

In response to this:

rebased to pick Red Hat Trusted App Pipeline CI fix
/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

enxebre · 2023-06-13T16:46:34Z

{Failed === RUN TestUpgradeControlPlane/EnsureNoCrashingPods
util.go:457: Container oauth-server in pod oauth-openshift-74bf787c59-6gh9p has a restartCount > 0 (2)

/test e2e-aws

openshift-ci-robot · 2023-06-13T17:58:44Z

/retest-required

Remaining retests: 0 against base HEAD af3de4f and 2 for PR HEAD 8093e27 in total

enxebre · 2023-06-13T22:20:11Z

/test images
/test e2e-kubevirt-aws-ovn

openshift-ci bot added the do-not-merge/needs-area label Jun 8, 2023

openshift-ci bot requested review from csrwng and sjenning June 8, 2023 16:47

openshift-ci bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-area labels Jun 8, 2023

enxebre mentioned this pull request Jun 8, 2023

Disable nodepool replace upgrade test #2665

Merged

4 tasks

csrwng approved these changes Jun 8, 2023

View reviewed changes

openshift-ci bot assigned csrwng Jun 8, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 8, 2023

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 8, 2023

enxebre force-pushed the feature-gate-nodepools branch from 7c2cc5e to 9126c6c Compare June 9, 2023 08:56

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 9, 2023

enxebre force-pushed the feature-gate-nodepools branch 4 times, most recently from aab9280 to ae768a3 Compare June 9, 2023 12:35

openshift-ci bot added the area/testing Indicates the PR includes changes for e2e testing label Jun 9, 2023

This was referenced Jun 9, 2023

OCPBUGS-13656: MCO-632: Update kube deps to 1.27.2 openshift/machine-config-operator#3735

Merged

OCPBUGS-14399: Minor fix to support protectKernelDefaults field in Kubelet Config openshift/machine-config-operator#3736

Merged

enxebre force-pushed the feature-gate-nodepools branch 3 times, most recently from b1cc626 to 31a5d2a Compare June 13, 2023 08:23

enxebre force-pushed the feature-gate-nodepools branch 4 times, most recently from 482658f to 07d031b Compare June 13, 2023 12:49

muraee reviewed Jun 13, 2023

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 13, 2023

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 13, 2023

enxebre force-pushed the feature-gate-nodepools branch from 07d031b to 8093e27 Compare June 13, 2023 15:33

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jun 13, 2023

enxebre added the lgtm Indicates that a PR is ready to be merged. label Jun 13, 2023

openshift-merge-robot merged commit f8a14a8 into openshift:main Jun 14, 2023
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Let payload provider render feature gate yaml #2664

Let payload provider render feature gate yaml #2664

enxebre commented Jun 8, 2023

csrwng left a comment

csrwng Jun 8, 2023

enxebre Jun 9, 2023 •

edited

openshift-ci bot commented Jun 8, 2023

csrwng commented Jun 8, 2023

enxebre commented Jun 9, 2023

enxebre commented Jun 9, 2023

netlify bot commented Jun 13, 2023 •

edited

enxebre commented Jun 13, 2023

JoelSpeed commented Jun 13, 2023

muraee Jun 13, 2023

enxebre Jun 13, 2023

muraee commented Jun 13, 2023

enxebre commented Jun 13, 2023 •

edited

csrwng commented Jun 13, 2023

enxebre commented Jun 13, 2023

openshift-ci bot commented Jun 13, 2023

enxebre commented Jun 13, 2023

openshift-ci bot commented Jun 13, 2023

enxebre commented Jun 13, 2023

openshift-ci-robot commented Jun 13, 2023

enxebre commented Jun 13, 2023

Let payload provider render feature gate yaml #2664

Let payload provider render feature gate yaml #2664

Conversation

enxebre commented Jun 8, 2023

csrwng left a comment

Choose a reason for hiding this comment

csrwng Jun 8, 2023

Choose a reason for hiding this comment

enxebre Jun 9, 2023 • edited

Choose a reason for hiding this comment

openshift-ci bot commented Jun 8, 2023

csrwng commented Jun 8, 2023

enxebre commented Jun 9, 2023

enxebre commented Jun 9, 2023

netlify bot commented Jun 13, 2023 • edited

✅ Deploy Preview for hypershift-docs ready!

enxebre commented Jun 13, 2023

JoelSpeed commented Jun 13, 2023

muraee Jun 13, 2023

Choose a reason for hiding this comment

enxebre Jun 13, 2023

Choose a reason for hiding this comment

muraee commented Jun 13, 2023

enxebre commented Jun 13, 2023 • edited

csrwng commented Jun 13, 2023

enxebre commented Jun 13, 2023

openshift-ci bot commented Jun 13, 2023

enxebre commented Jun 13, 2023

openshift-ci bot commented Jun 13, 2023

enxebre commented Jun 13, 2023

openshift-ci-robot commented Jun 13, 2023

enxebre commented Jun 13, 2023

enxebre Jun 9, 2023 •

edited

netlify bot commented Jun 13, 2023 •

edited

enxebre commented Jun 13, 2023 •

edited