OCPBUGS-29290: AWS: Always persist the existing node name on 4.14

[JoelSpeed]

On AWS, we persist the hostname into a file on disk, with the idea being that, this value is passed as a --hostname-override to kubelet, such that it should register with the overriden hostname to the cluster.
In 4.13 and lower, if the hostname override is set, it is ignored as the AWS cloud provider integration takes precedence.

To ensure the upgrade from 4.13 to 4.14 is smooth (where the in-tree logic is removed), we must persist the hostname that the node originally registered with. If we do not, then the host name may change between versions, which breaks the upgrade and requires manual intervention to recover the nodes by approving new certificates, and removing the old node objects.

[openshift-ci-robot]

@JoelSpeed: This pull request references Jira Issue OCPBUGS-29290, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.13.z) matches configured target version for branch (4.13.z)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
• dependent bug Jira Issue OCPBUGS-29916 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-29916 targets the "4.14.z" version, which is one of the valid target versions: 4.14.0, 4.14.z
• bug has dependents

Requesting review from QA contact:
/cc @sunzhaohua2

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

On AWS, we persist the hostname into a file on disk, with the idea being that, this value is passed as a --hostname-override to kubelet, such that it should register with the overriden hostname to the cluster.
In 4.13 and lower, if the hostname override is set, it is ignored as the AWS cloud provider integration takes precedence.

To ensure the upgrade from 4.13 to 4.14 is smooth (where the in-tree logic is removed), we must persist the hostname that the node originally registered with. If we do not, then the host name may change between versions, which breaks the upgrade and requires manual intervention to recover the nodes by approving new certificates, and removing the old node objects.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[JoelSpeed]

/payload-job periodic-ci-openshift-release-master-nightly-4.14-upgrade-from-stable-4.13-e2e-aws-sdn-upgrade

[openshift-ci]

@JoelSpeed: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-nightly-4.14-upgrade-from-stable-4.13-e2e-aws-sdn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/003408b0-d589-11ee-8c09-fd4f3aba27c1-0

[openshift-ci-robot]

@JoelSpeed: This pull request references Jira Issue OCPBUGS-29290, which is invalid:

• expected the bug to target either version "4.14." or "openshift-4.14.", but it targets "4.13.z" instead
• expected dependent Jira Issue OCPBUGS-29916 to target a version in 4.15.0, 4.15.z, but it targets "4.14.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

On AWS, we persist the hostname into a file on disk, with the idea being that, this value is passed as a --hostname-override to kubelet, such that it should register with the overriden hostname to the cluster.
In 4.13 and lower, if the hostname override is set, it is ignored as the AWS cloud provider integration takes precedence.

To ensure the upgrade from 4.13 to 4.14 is smooth (where the in-tree logic is removed), we must persist the hostname that the node originally registered with. If we do not, then the host name may change between versions, which breaks the upgrade and requires manual intervention to recover the nodes by approving new certificates, and removing the old node objects.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[JoelSpeed]

/jira refresh

[openshift-ci-robot]

@JoelSpeed: This pull request references Jira Issue OCPBUGS-29290, which is valid.

6 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.14.z) matches configured target version for branch (4.14.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• dependent bug Jira Issue OCPBUGS-29916 is in the state Closed (Done), which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-29916 targets the "4.15.z" version, which is one of the valid target versions: 4.15.0, 4.15.z
• bug has dependents

Requesting review from QA contact:
/cc @huali9

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[huali9]

/label qe-approved

[JoelSpeed]

/retest

[damdo]

Bash syntax and reasoning
/lgtm

Is there an automated test in CI that exercises this scenario or did you test this manually somehow?
Thanks

[JoelSpeed]

@damdo I have manually tested yesterday on my own cluster and @huali9 has also tested again this morning, detailed test report in the attached bug

[theobarberbany]

possible silly Q / nit

why are we checking that the hostname is non empty here? is it possible that the extraction above could fail? do we want explicit error handling in that case? or will
if [[ ! -z "${HOSTNAME}" ]]; then skipping the following code be sufficient?

[JoelSpeed]

I expected it may be possible that the file had been created but was empty, in that case, I just wanted to fallback to the original logic, which I believe is what will happen given this currently

[theobarberbany]

one question, but other than that

/lgtm

[sinnykumari]

/label backport-risk-assessed

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: damdo, JoelSpeed, sinnykumari, theobarberbany

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sinnykumari]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sergiordlr]

/label cherry-pick-approved

[openshift-ci]

@JoelSpeed: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-upgrade	e4ccde8	link	true	/test e2e-aws-ovn-upgrade
ci/prow/e2e-aws-ovn-fips	e4ccde8	link	false	/test e2e-aws-ovn-fips
ci/prow/e2e-gcp-rt	e4ccde8	link	false	/test e2e-gcp-rt
ci/prow/okd-scos-images	e4ccde8	link	true	/test okd-scos-images
ci/prow/images	e4ccde8	link	true	/test images
ci/prow/verify	e4ccde8	link	true	/test verify
ci/prow/okd-images	e4ccde8	link	true	/test okd-images
ci/prow/e2e-aws-ovn	e4ccde8	link	true	/test e2e-aws-ovn
ci/prow/bootstrap-unit	e4ccde8	link	false	/test bootstrap-unit
ci/prow/okd-scos-e2e-aws-ovn	e4ccde8	link	false	/test okd-scos-e2e-aws-ovn
ci/prow/e2e-aws-ovn-fips-op	e4ccde8	link	false	/test e2e-aws-ovn-fips-op

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-ci-robot]

@JoelSpeed: Jira Issue OCPBUGS-29290: All pull requests linked via external trackers have merged:

• openshift/machine-config-operator#4215

Jira Issue OCPBUGS-29290 has been moved to the MODIFIED state.

In response to this:

On AWS, we persist the hostname into a file on disk, with the idea being that, this value is passed as a --hostname-override to kubelet, such that it should register with the overriden hostname to the cluster.
In 4.13 and lower, if the hostname override is set, it is ignored as the AWS cloud provider integration takes precedence.

To ensure the upgrade from 4.13 to 4.14 is smooth (where the in-tree logic is removed), we must persist the hostname that the node originally registered with. If we do not, then the host name may change between versions, which breaks the upgrade and requires manual intervention to recover the nodes by approving new certificates, and removing the old node objects.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-machine-config-operator-container-v4.14.0-202402291242.p0.g8f8dbba.assembly.stream.el8 for distgit ose-machine-config-operator.
All builds following this will include this PR.

[openshift-merge-robot]

Fix included in accepted release 4.14.0-0.nightly-2024-02-29-134959