MCO-497: Prep node for RHCOS 9 upgrade #3553

cheesesashimi · 2023-02-13T18:51:08Z

- What I did

- How to verify it

- Description for the changelog

openshift-ci · 2023-02-13T18:52:03Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2023-02-13T18:54:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cheesesashimi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cheesesashimi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2023-02-13T19:54:29Z

@cheesesashimi: This pull request references MCO-497 which is a valid jira issue.

In response to this:

- What I did

- How to verify it

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cgwalters · 2023-02-13T23:11:48Z

PR to add hotfixes up at coreos/coreos-assembler#3352 and openshift/os#1162

Demonstration container up at quay.io/cgwalters/ostest:

$ podman run --rm -ti --entrypoint ls quay.io/cgwalters/ostest -al /usr/share/rpm-ostree/extensions/hotfixes
total 2936
drwxr-xr-x. 1 root root      94 Feb 13 23:06 .
drwxr-xr-x. 1 root root      16 Feb 13 23:06 ..
-rw-r--r--. 1 root root      94 Feb 13 23:03 hotfixes.json
-rw-r--r--. 1 root root 3002316 Feb 13 23:03 rpm-ostree-2022.2-2.el8.x86_64.rpm

cheesesashimi · 2023-02-14T17:30:59Z

pkg/daemon/update.go

-	} else if err := dn.NodeUpdaterClient.RebaseLayered(newURL); err != nil {
+	}
+
+	if err := dn.prepForRHCOS9(newURL); err != nil {


I'm fairly certain this is correct for the applyLayeredOSChanges() path. However, I'm not as certain for the applyLegacyOSChanges() path.

I think applyLegacyOSChanges() may be fully dead code now? cc @jkyros

If that path is dead, I'd like to remove it in a follow-up PR. No sense in keeping it around if it's not being used.

applyLegacyOSChanges() can only be reached if someone feeds us an "old format" image. I'm assuming the only way to get to 4.13 is through a 4.12 version (like, we don't support someone forcing an upgrade from 4.11 to 4.13), so yes, if that assumption is true, I think we can safely take applyLegacyOSChanges() out for 4.13, which is awesome, because that was an unpleasant bit of code. 😄

cheesesashimi · 2023-02-14T17:33:02Z

pkg/daemon/osrelease/osrelease.go

+	}
+
+	// Only FCOS and SCOS set this field.
+	if osName, osNameOK := imageLabels["io.openshift.build.version-display-names"]; osNameOK {


It seems odd that only FCOS and SCOS set this field and that RHCOS simply doesn't. It is possible that I examined the wrong container labels when I was setting up the tests for this code.

Yeah 😢 This is because of the machine-os-content zombie issue...some links for this in openshift/os#1048

Basically oc adm release new barfs and dies if there are two container images in the release which claim to be an operating system. OKD already set things up so that machine-os-content is an alias, so they don't have this problem.

cheesesashimi · 2023-02-14T17:35:07Z

pkg/daemon/osrelease/osrelease_test.go

+func TestOSReleaseFromImageLabels(t *testing.T) {
+	t.Parallel()
+
+	// $ skopeo inspect --no-tags "docker://$(oc adm release info --pullspecs "quay.io/openshift-release-dev/ocp-release:4.12.3-x86_64" -o json | jq -r '.references.spec.tags[] | select(.name == "rhel-coreos-8") | .from.name')" | jq '.Labels'


I've included the commands I used to obtain the image labels for a sanity check. If I was looking at the wrong source image, please let me know.

cheesesashimi · 2023-02-14T17:37:39Z

test/e2e/node_os_test.go

 func TestOSDetection(t *testing.T) {
 	cs := framework.NewClientSet("")
 	nodes, err := cs.CoreV1Interface.Nodes().List(context.TODO(), metav1.ListOptions{})
 	require.NoError(t, err)

+	osImageRelease, osImageReleaseSrc := getOSReleaseInfoForOSImage(t, cs, nodes.Items[0])


I added an additional check to this e2e test that gets the OS image labels given the current release on the test cluster. This way, we can get the same early warning if something changes unexpectedly.

cheesesashimi · 2023-02-14T17:42:47Z

pkg/daemon/update.go

+
+	// Upgrades rpm-ostree.
+	// TODO(zzlotnik): Is this the right command to upgrade rpm-ostree?
+	if _, err := runGetOut("rpm", "-Uvh", "rpm-ostree"); err != nil {


Is this the right way to upgrade rpm-ostree with the extensions container repo set up?

One tricky bit is we'll likely need to run this code in the host mount namespace. So something like:

systemd-run -qP /bin/sh -c 'rpm-ostree usroverlay; rpm -Uvh /path/to/extensions/hotfixes/8/*.rpm && systemctl restart rpm-ostreed'

or so.

The actual path to the mounted hotfixes will be a hotfixes subdirectory of the extensions path.

Gotcha. Yeah, that may take a bit to get right or I'll have to drop it in a temp file first.

cheesesashimi · 2023-02-14T18:29:07Z

/test unit
/test verify

openshift-ci · 2023-02-14T18:32:31Z

@cheesesashimi: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

cgwalters · 2023-02-14T22:06:19Z

Zack and I had a realtime chat about this. Summary from my PoV:

Get https://bugzilla.redhat.com/show_bug.cgi?id=2169429 at least QE approved and then we'll fast-track it into the OCP 4.13/rhel8 channel
Update Add hotfixes-rhel-coreos-9.yaml os#1162 to pull in that fix
Rerun a build of the 4.13-9.2 pipeline to pull in an updated extensions container with that patched rpm-ostree
Clean up and iterate on this PR, land it
Rebase Switch to rhel-coreos-9 #3485 on master

or so?

cgwalters · 2023-02-14T22:10:07Z

pkg/daemon/osrelease/osrelease.go

+
+	version := ""
+	versionOK := false
+	if version, versionOK = imageLabels["version"]; !versionOK {


Let's also try to handle the standard OCI label for this. See ostreedev/ostree-rs-ext#454

Basically the standard key is org.opencontainers.image.version so let's look for that first.

I don't see that key in any of the labels I used to set up the tests. Did I not get the right labels?

cgwalters · 2023-02-14T22:13:06Z

pkg/daemon/update.go

+
+	// Upgrades rpm-ostree.
+	// TODO(zzlotnik): Is this the right command to upgrade rpm-ostree?
+	if _, err := runGetOut("rpm", "-Uvh", "rpm-ostree"); err != nil {


One tricky bit is we'll likely need to run this code in the host mount namespace. So something like:

systemd-run -qP /bin/sh -c 'rpm-ostree usroverlay; rpm -Uvh /path/to/extensions/hotfixes/8/*.rpm && systemctl restart rpm-ostreed'

or so.

The actual path to the mounted hotfixes will be a hotfixes subdirectory of the extensions path.

cgwalters · 2023-02-14T22:15:24Z

pkg/daemon/update.go

-	} else if err := dn.NodeUpdaterClient.RebaseLayered(newURL); err != nil {
+	}
+
+	if err := dn.prepForRHCOS9(newURL); err != nil {


I think applyLegacyOSChanges() may be fully dead code now? cc @jkyros

cheesesashimi · 2023-02-15T15:00:05Z

Closing due to strategy change. See: https://issues.redhat.com/browse/OCPBUGS-7275?focusedCommentId=21743230&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-21743230

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 13, 2023

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 13, 2023

cheesesashimi force-pushed the zzlotnik/upgrade-rpmostree-first branch from 235b52e to e7b4613 Compare February 13, 2023 19:54

cheesesashimi changed the title ~~Prep node for RHCOS 9 upgrade~~ MCO-497: Prep node for RHCOS 9 upgrade Feb 13, 2023

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Feb 13, 2023

cheesesashimi force-pushed the zzlotnik/upgrade-rpmostree-first branch 2 times, most recently from 65d6d64 to 166663d Compare February 13, 2023 21:16

infer OS release from OS image labels

273df5e

cheesesashimi force-pushed the zzlotnik/upgrade-rpmostree-first branch from 7a53c11 to ac92a76 Compare February 14, 2023 16:07

inspect OS image labels during e2e for early warning

517097d

cheesesashimi force-pushed the zzlotnik/upgrade-rpmostree-first branch 2 times, most recently from 9dda49f to 6697fc4 Compare February 14, 2023 17:19

use extension container repo to upgrade rpm-ostree

5d97232

cheesesashimi force-pushed the zzlotnik/upgrade-rpmostree-first branch from 6697fc4 to 5d97232 Compare February 14, 2023 17:27

cheesesashimi commented Feb 14, 2023

View reviewed changes

cgwalters reviewed Feb 14, 2023

View reviewed changes

osrelease: handle additional edge-cases

9e395be

cheesesashimi closed this Feb 15, 2023

cheesesashimi deleted the zzlotnik/upgrade-rpmostree-first branch March 21, 2024 14:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MCO-497: Prep node for RHCOS 9 upgrade #3553

MCO-497: Prep node for RHCOS 9 upgrade #3553

cheesesashimi commented Feb 13, 2023

openshift-ci bot commented Feb 13, 2023

openshift-ci bot commented Feb 13, 2023

openshift-ci-robot commented Feb 13, 2023 •

edited by openshift-ci bot

cgwalters commented Feb 13, 2023

cheesesashimi Feb 14, 2023

cgwalters Feb 14, 2023

cheesesashimi Feb 15, 2023

jkyros Feb 15, 2023 •

edited

cheesesashimi Feb 14, 2023

cgwalters Feb 14, 2023 •

edited

cheesesashimi Feb 14, 2023

cheesesashimi Feb 14, 2023

cheesesashimi Feb 14, 2023

cgwalters Feb 14, 2023

cheesesashimi Feb 15, 2023

cheesesashimi commented Feb 14, 2023

openshift-ci bot commented Feb 14, 2023

cgwalters commented Feb 14, 2023

cgwalters Feb 14, 2023

cheesesashimi Feb 14, 2023

cgwalters Feb 14, 2023

cgwalters Feb 14, 2023

cheesesashimi commented Feb 15, 2023

MCO-497: Prep node for RHCOS 9 upgrade #3553

MCO-497: Prep node for RHCOS 9 upgrade #3553

Conversation

cheesesashimi commented Feb 13, 2023

openshift-ci bot commented Feb 13, 2023

openshift-ci bot commented Feb 13, 2023

openshift-ci-robot commented Feb 13, 2023 • edited by openshift-ci bot

cgwalters commented Feb 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkyros Feb 15, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgwalters Feb 14, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheesesashimi commented Feb 14, 2023

openshift-ci bot commented Feb 14, 2023

cgwalters commented Feb 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cheesesashimi commented Feb 15, 2023

openshift-ci-robot commented Feb 13, 2023 •

edited by openshift-ci bot

jkyros Feb 15, 2023 •

edited

cgwalters Feb 14, 2023 •

edited