Produce official ppc64le architecture artifacts

[mkumatag]

Describe the enhancement

This issue for requesting ppc64le platform support for the Fedora CoreOS. Recently arm64 support has been added via #13, this issue forked from the #13 (comment) discussion.

cc @travier

[dustymabe]

crosslinking coreos/coreos-assembler#2473 here since there is some good discussion there about some challenges ahead.

[dustymabe]

enablement for this has landed in a few PRs:

• multi-arch-builders: add ppc64le builder info fedora-coreos-pipeline#597
• jobs/build-cosa: build COSA container for ppc64le fedora-coreos-pipeline#601
• only set LIBGUESTFS_HV for ppc64le on el7 coreos-assembler#3033
• browser: show ppc64le built artifacts fedora-coreos-browser#38
• enable ppc64le in streams.groovy fedora-coreos-pipeline#609

and some pending:

• graph-builder: add scopes for ppc64le fedora-coreos-cincinnati#84

[dustymabe]

Marking this as complete as it should land in next week's releases.

[dustymabe]

Actually, we probably need to try to make sure we get coreos/fedora-coreos-config#1844 in on Monday before we start the releases. On my existing builder I'm short on space in /boot:

[core@buildvm-ppc64le-fcos01 srv]$ rpm-ostree status 
State: idle
Deployments:
● ostree-unverified-image:oci-archive:/srv/fedora-coreos-36.20220809.20.3-ostree.ppc64le.ociarchive
                   Digest: sha256:98ed9b1277677bff0455a937c85816ebd3431d38dcebe2bfbdfb66ee7646c0e0
                  Version: 36.20220809.20.3 (2022-08-10T20:09:39Z)

  ostree-unverified-image:oci-archive:/srv/fedora-coreos-36.20220711.dev.0-ostree.ppc64le.ociarchive
                   Digest: sha256:44122282e785ad742350b3862ebbffb5d60a1c8ab4168237c27d4d1efaeb94c3
                  Version: 36.20220711.dev.0 (2022-07-11T08:20:40Z)
[core@buildvm-ppc64le-fcos01 srv]$ 
[core@buildvm-ppc64le-fcos01 srv]$ df -kh /boot/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       362M  248M   92M  74% /boot
[core@buildvm-ppc64le-fcos01 srv]$ 
[core@buildvm-ppc64le-fcos01 srv]$ sudo du -sh /boot/ostree/*
121M    /boot/ostree/fedora-coreos-4629dab1861e40544bb601f84b5fd6ab6fc20a223e76df751bb67d3a2620bbaf
121M    /boot/ostree/fedora-coreos-49d9a2f5d0cc617e86c58c895447654c1c5d563384ee96affba8ad5a1bc019e4

[jlebon]

[core@buildvm-ppc64le-fcos01 srv]$ df -kh /boot/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       362M  248M   92M  74% /boot
[core@buildvm-ppc64le-fcos01 srv]$ 
[core@buildvm-ppc64le-fcos01 srv]$ sudo du -sh /boot/ostree/*
121M    /boot/ostree/fedora-coreos-4629dab1861e40544bb601f84b5fd6ab6fc20a223e76df751bb67d3a2620bbaf
121M    /boot/ostree/fedora-coreos-49d9a2f5d0cc617e86c58c895447654c1c5d563384ee96affba8ad5a1bc019e4

Right, this is worse on ppc64le (one of the reasons #1247 was filed was for that arch, where nodes actually did hit ENOSPC). Although we'll have to deal with that in RHCOS for existing machines, for FCOS we have an opportunity to start off ppc64le with a larger /boot already (and we really should've done that for the recently added arches too).

Maybe let's add support for configuring it via image.yaml? We'll need a knob like that anyway to ratchet changing the boot partition size.

[dustymabe]

for FCOS we have an opportunity to start off ppc64le with a larger /boot already (and we really should've done that for the recently added arches too).

yeah I'm fine with that as long as we have some concensus on what we want things to look like for all architectures before we make any changes here.

I'll try to hold this until we make that decision.

[dustymabe]

Update: coreos/fedora-coreos-config#1844 landed. Here is what I see on our ppc64le builder now (after deploying 36.20220916.20.1):

$ pwd
/boot/ostree
$ sudo du -sh ./* 
112M    ./fedora-coreos-39dc2f20c9a2c1a9a427fd7a0bc456956f3d90c6ed96beeb946c24779e0af338
121M    ./fedora-coreos-d02361e161cc8b1bc393f3686b403f3d5f4fb587853234e1139c1f0d9db38aba

$ file ./fedora-coreos-39dc2f20c9a2c1a9a427fd7a0bc456956f3d90c6ed96beeb946c24779e0af338/initramfs-5.19.8-200.fc36.ppc64le.img 
./fedora-coreos-39dc2f20c9a2c1a9a427fd7a0bc456956f3d90c6ed96beeb946c24779e0af338/initramfs-5.19.8-200.fc36.ppc64le.img: Zstandard compressed data (v0.8+), Dictionary ID: None


$ file ./fedora-coreos-d02361e161cc8b1bc393f3686b403f3d5f4fb587853234e1139c1f0d9db38aba/initramfs-5.18.19-200.fc36.ppc64le.img 
./fedora-coreos-d02361e161cc8b1bc393f3686b403f3d5f4fb587853234e1139c1f0d9db38aba/initramfs-5.18.19-200.fc36.ppc64le.img: gzip compressed data, max compression, from Unix, original size modulo 2^32 1024

$ df -kh /boot/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       362M  239M  101M  71% /boot

[dustymabe]

Here's a summary of what's blocking this right now: #1247 (comment)

UPDATE: The space issue in boot ended up being because the kernel is uncompressed, not because the initramfs file was larger: #1247 (comment)

[dustymabe]

As a group we have decided to ultimately increase the size of our /boot partition. We should consider releasing ppc64le with the new partition size (once we know what that will be) from the start.

[dustymabe]

Now that we have OSTree early/auto pruning enabled for ppc64le we should be able to deal effectively with the /boot space issue without having to block on #1465. We have enabled ppc64le for testing and next in coreos/fedora-coreos-pipeline#876 and will produce official artifacts in the next round of releases.

[dustymabe]

The fix for this went into next stream release 38.20230609.1.0. Please try out the new release and report issues.

[dustymabe]

The fix for this went into testing stream release 38.20230609.2.1. Please try out the new release and report issues.

[dustymabe]

The fix for this went into stable stream release 38.20230625.3.0.