Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RHCOS 4.11 x86 - Found kernel warning (arch/x86/kernel/apic/apic.c:1547 setup_local_APIC+0x269/0x370 ) #862

Closed
miabbott opened this issue Jun 22, 2022 · 20 comments
Labels

Comments

@miabbott
Copy link
Member

Seen during kola -p qemu-unpriv --output-dir tmp/kola run --qemu-firmware=uefi basic:

kola -p qemu-unpriv --output-dir tmp/kola run --qemu-firmware=uefi basic
������  Skipping kola test pattern "coreos.ignition.journald-log":
  ���� https://github.com/coreos/coreos-assembler/issues/1173
=== RUN   basic
=== RUN   basic/ReadOnly
=== RUN   basic/Useradd
=== RUN   basic/MachineID
=== RUN   basic/RHCOSGrowpart
=== RUN   basic/PortSSH
=== RUN   basic/DbusPerms
=== RUN   basic/ServicesActive
--- FAIL: basic (63.23s)
    --- PASS: basic/ReadOnly (0.09s)
    --- PASS: basic/Useradd (0.38s)
    --- PASS: basic/MachineID (0.10s)
    --- PASS: basic/RHCOSGrowpart (0.11s)
    --- PASS: basic/PortSSH (0.09s)
    --- PASS: basic/DbusPerms (0.26s)
    --- PASS: basic/ServicesActive (0.11s)
        harness.go:1370: Found kernel warning (arch/x86/kernel/apic/apic.c:1547 setup_local_APIC+0x269/0x370
) on machine 740a0505-1cde-4245-990f-332e7cf6c318 console
FAIL, output in tmp/kola

In journal.txt

Jun 22 14:51:51.586854 kernel: Booting paravirtualized kernel on KVM                                                                                           
Jun 22 14:51:51.586859 kernel: clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
Jun 22 14:51:51.586865 kernel: setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
Jun 22 14:51:51.586870 kernel: percpu: Embedded 55 pages/cpu s188416 r8192 d28672 u2097152
Jun 22 14:51:51.586875 kernel: pcpu-alloc: s188416 r8192 d28672 u2097152 alloc=1*2097152
Jun 22 14:51:51.586881 kernel: pcpu-alloc: [0] 0 
Jun 22 14:51:51.586886 kernel: kvm-guest: stealtime: cpu 0, msr 3ce2d080
Jun 22 14:51:51.586891 kernel: kvm-guest: PV spinlocks disabled, single CPU
Jun 22 14:51:51.586896 kernel: Built 1 zonelists, mobility grouping on.  Total pages: 217618
Jun 22 14:51:51.586902 kernel: Policy zone: DMA32
Jun 22 14:51:51.586910 kernel: Kernel command line: BOOT_IMAGE=(hd0,gpt3)/ostree/rhcos-918339bcdab38512ab2e0d8c87af5c8f7e285f541200d935a9e2c15066ac6327/vmlinuz-4.18.0-372.9.1.el8.x86_64 random.trust_cpu=on ignition.platform.id=qemu console=tty0 console=ttyS0,115200n8 ignition.firstboot ostree=/ostree/boot.1/rhcos/918
339bcdab38512ab2e0d8c87af5c8f7e285f541200d935a9e2c15066ac6327/0
Jun 22 14:51:51.586916 kernel: Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com.
Jun 22 14:51:51.586921 kernel: Memory: 196584K/892220K available (12293K kernel code, 5865K rwdata, 8292K rodata, 2520K init, 14348K bss, 197316K reserved, 0K cma-reserved)
Jun 22 14:51:51.586927 kernel: SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Jun 22 14:51:51.586932 kernel: Kernel/User page tables isolation: enabled
Jun 22 14:51:51.586937 kernel: ftrace: allocating 39999 entries in 157 pages
Jun 22 14:51:51.586942 kernel: ftrace: allocated 157 pages with 5 groups
Jun 22 14:51:51.586947 kernel: rcu: Hierarchical RCU implementation.
Jun 22 14:51:51.586953 kernel: rcu:         RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.
Jun 22 14:51:51.586958 kernel:         Rude variant of Tasks RCU enabled.
Jun 22 14:51:51.586963 kernel:         Tracing variant of Tasks RCU enabled.
Jun 22 14:51:51.586971 kernel: rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
Jun 22 14:51:51.586976 kernel: rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
Jun 22 14:51:51.586981 kernel: NR_IRQS: 524544, nr_irqs: 256, preallocated irqs: 16
Jun 22 14:51:51.586986 kernel: random: crng done (trusting CPU's manufacturer)
Jun 22 14:51:51.586991 kernel: Console: colour dummy device 80x25
Jun 22 14:51:51.586997 kernel: printk: console [tty0] enabled
Jun 22 14:51:51.587002 kernel: printk: console [ttyS0] enabled
Jun 22 14:51:51.587007 kernel: ACPI: Core revision 20210604
Jun 22 14:51:51.587013 kernel: clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
Jun 22 14:51:51.587018 kernel: hpet clockevent registered
Jun 22 14:51:51.587023 kernel: APIC: Switch to symmetric I/O mode setup
Jun 22 14:51:51.587031 kernel: x2apic enabled
Jun 22 14:51:51.587036 kernel: Switched APIC routing to physical x2apic.
Jun 22 14:51:51.587041 kernel: WARNING: CPU: 0 PID: 0 at arch/x86/kernel/apic/apic.c:1547 setup_local_APIC+0x269/0x370
Jun 22 14:51:51.587047 kernel: Modules linked in:
Jun 22 14:51:51.587053 kernel: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.18.0-372.9.1.el8.x86_64 #1
Jun 22 14:51:51.587058 kernel: Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
Jun 22 14:51:51.587063 kernel: RIP: 0010:setup_local_APIC+0x269/0x370
Jun 22 14:51:51.587069 kernel: Code: 74 2f 8b 0d bd ef d4 01 85 c9 74 25 0f 31 c1 e1 0a 48 c1 e2 20 48 01 d9 48 09 c2 48 29 d1 49 89 cd 4d 85 ed 0f 8f 5b fe ff ff <0f> 0b e9 10 ff ff ff 49 83 ed 01 eb ea 0f 31 48 c1 e2 20 48 89 d3
Jun 22 14:51:51.587074 kernel: RSP: 0000:ffffffff88803eb8 EFLAGS: 00010286
Jun 22 14:51:51.587079 kernel: RAX: 0000000007d56564 RBX: 00000011636cc0d0 RCX: fffffffffffffb6c
Jun 22 14:51:51.587086 kernel: RDX: 0000001207d56564 RSI: 0000000000000020 RDI: ffffffff88803ec8
Jun 22 14:51:51.587091 kernel: RBP: 0000000000000000 R08: ffffffff88803ee8 R09: ffffffff88803eec
Jun 22 14:51:51.587096 kernel: R10: 0000000000000001 R11: ffffffff88803d18 R12: 00000000000000f0
Jun 22 14:51:51.587101 kernel: R13: fffffffffffffb6c R14: 0000000000000020 R15: 0000000000000020
Jun 22 14:51:51.587107 kernel: FS:  0000000000000000(0000) GS:ffff9ccbbce00000(0000) knlGS:0000000000000000
Jun 22 14:51:51.587112 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jun 22 14:51:51.587117 kernel: CR2: ffff9ccb8e001000 CR3: 000000000c610001 CR4: 00000000000606b0
Jun 22 14:51:51.587122 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jun 22 14:51:51.587127 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jun 22 14:51:51.587132 kernel: Call Trace:
Jun 22 14:51:51.587140 kernel:  apic_bsp_setup+0x62/0x80
Jun 22 14:51:51.587145 kernel:  x86_late_time_init+0x29/0x39
Jun 22 14:51:51.587150 kernel:  start_kernel+0x486/0x542
Jun 22 14:51:51.587155 kernel:  secondary_startup_64_no_verify+0xc2/0xcb
Jun 22 14:51:51.587160 kernel: ---[ end trace 02abcd4e6b23d7dc ]---

Interestingly, the VM under test successfully made it to the login prompt after this, so it doesn't look like a fatal error for the VM, just for the test harness.

Notably this occurred after the coreos-assembler:main image started to use F36 as an outcome of #861

@miabbott
Copy link
Member Author

console.txt
journal.txt

@miabbott
Copy link
Member Author

Unfortunately, this looks like it is reproducible and is impacting our ability to produce RHCOS 4.11 builds.

@ashcrow ashcrow added the jira label Jun 22, 2022
@miabbott
Copy link
Member Author

miabbott commented Jun 22, 2022

My gut says this is related to running F36 binaries in a container on the RHEL7 host where the downstream RHCOS pipelines are running in privileged PSI. (More specifically a RHEL8 guest via F36 qemu on a RHEL7 host)

One can try to spin up a cosa pod in privileged PSI to see this is reproducible manually. Then maybe build an F35 based cosa and repeat the test with that.

@miabbott
Copy link
Member Author

I was able to reproduce this by hand using a cosa pod as described above. Though I saw the failure in the uefi-secure branch:

kola -p qemu-unpriv --output-dir tmp/kola run --qemu-firmware=uefi-secure basic
⚠️  Skipping kola test pattern "coreos.ignition.journald-log":
  👉 https://github.com/coreos/coreos-assembler/issues/1173
=== RUN   basic
=== RUN   basic/ReadOnly
=== RUN   basic/Useradd
=== RUN   basic/MachineID
=== RUN   basic/RHCOSGrowpart
=== RUN   basic/PortSSH
=== RUN   basic/DbusPerms
=== RUN   basic/ServicesActive
--- FAIL: basic (66.58s)
    --- PASS: basic/ReadOnly (0.09s)
    --- PASS: basic/Useradd (0.38s)
    --- PASS: basic/MachineID (0.10s)
    --- PASS: basic/RHCOSGrowpart (0.10s)
    --- PASS: basic/PortSSH (0.09s)
    --- PASS: basic/DbusPerms (0.26s)
    --- PASS: basic/ServicesActive (0.11s)
) on machine 80597065-d2e9-4a46-8b6f-63f363469394 consoleernel/apic/apic.c:1547 setup_local_APIC+0x269/0x370
FAIL, output in tmp/kola

@miabbott
Copy link
Member Author

Leaving a breadcrumb for future me:

Failed to pull image "quay.io/miabbott/coreos-assembler:f35": rpc error: code = Unknown desc = missing signature key

...means that the image pushed to Quay was in OCI format and not in v2s2 format

podman push -f v2s2 ... will solve the issue

@miabbott
Copy link
Member Author

Building and testing with an F35 based coreos-assembler seems to work, at least for the --basic-qemu-scenarios use case.

Still running the full cosa kola run to see if all is well.

I'm going to recommend we temporarily build + test RHCOS 4.11 with an F35 based coreos-assembler in order to continue to provide updates for the rest of the OpenShift org.

@miabbott
Copy link
Member Author

Recording the diff of the RPMs in an F35 cosa vs. F36 cosa for additional context:

--- f35-cosa-rpms.txt	2022-06-22 16:16:54.875195749 -0400
+++ f36-cosa-rpms.txt	2022-06-22 16:17:12.319165574 -0400
@@ -1,4 +1,5 @@
-NetworkManager-libnm-1.32.12
+ModemManager-glib-1.18.8
+NetworkManager-libnm-1.38.0
 SDL2-2.0.22
 SDL2_image-2.0.5
 ShellCheck-0.7.2
@@ -6,27 +7,30 @@
 abattis-cantarell-fonts-0.301
 acl-2.3.1
 adobe-source-code-pro-fonts-2.030.1.050
-adwaita-cursor-theme-41.0
-adwaita-icon-theme-41.0
-alsa-lib-1.2.6.1
+adwaita-cursor-theme-42.0
+adwaita-icon-theme-42.0
+alsa-lib-1.2.7
 alternatives-1.19
-annobin-docs-10.66
-annobin-plugin-gcc-10.66
+annobin-docs-10.73
+annobin-plugin-gcc-10.73
 ansible-srpm-macros-1
 apr-1.7.0
 apr-util-1.6.1
 apr-util-bdb-1.6.1
 apr-util-openssl-1.6.1
 at-spi2-atk-2.38.0
-at-spi2-core-2.42.0
-atk-2.36.0
+at-spi2-core-2.44.1
+atk-2.38.0
 attr-2.5.1
 audit-libs-3.0.8
 augeas-libs-1.13.0
+authselect-1.4.0
+authselect-libs-1.4.0
+avahi-glib-0.8
 avahi-libs-0.8
-awscli-1.22.50
+awscli-1.22.88
 basesystem-11
-bash-5.1.8
+bash-5.1.16
 binutils-2.37
 binutils-gold-2.37
 bluez-libs-5.64
@@ -34,32 +38,31 @@
 boost-system-1.76.0
 boost-thread-1.76.0
 bootupd-0.2.6
-brlapi-0.8.2
-brltty-6.3
+brlapi-0.8.3
+brltty-6.4
 bsdtar-3.5.3
 btrfs-progs-5.18
 bubblewrap-0.5.0
-buildah-1.23.4
+buildah-1.26.0
 butane-0.14.0
 bzip2-1.0.8
 bzip2-libs-1.0.8
 ca-certificates-2021.2.52
-cairo-1.17.4
-cairo-gobject-1.17.4
+cairo-1.17.6
+cairo-gobject-1.17.6
 capstone-4.0.2
 catatonit-0.1.7
 cdparanoia-libs-10.2
 checkpolicy-3.3
-colord-libs-1.4.5
+colord-libs-1.4.6
 conmon-2.1.0
 container-selinux-2.187.0
-containernetworking-plugins-1.1.0
 containers-common-1
 coreos-installer-0.14.0
-coreutils-8.32
-coreutils-common-8.32
+coreutils-9.0
+coreutils-common-9.0
 cpio-2.13
-cpp-11.3.1
+cpp-12.1.1
 cracklib-2.9.6
 cracklib-dicts-2.9.6
 createrepo_c-0.20.0
@@ -67,28 +70,27 @@
 criu-3.17
 criu-libs-3.17
 crun-1.4.5
-crypto-policies-20210819
-crypto-policies-scripts-20210819
+crypto-policies-20220428
+crypto-policies-scripts-20220428
 cryptsetup-2.4.3
 cryptsetup-libs-2.4.3
-cups-libs-2.3.3op2
-curl-7.79.1
+cups-libs-2.4.2
+curl-7.82.0
 cyrus-sasl-2.1.27
 cyrus-sasl-gssapi-2.1.27
 cyrus-sasl-lib-2.1.27
 daxctl-libs-73
-dbus-1.12.22
-dbus-broker-29
-dbus-common-1.12.22
-dbus-libs-1.12.22
+dbus-1.14.0
+dbus-broker-31
+dbus-common-1.14.0
+dbus-libs-1.14.0
 dconf-0.40.0
 debugedit-5.0
-dejavu-sans-fonts-2.37
-deltarpm-3.6.2
 device-mapper-1.02.175
 device-mapper-event-1.02.175
 device-mapper-event-libs-1.02.175
 device-mapper-libs-1.02.175
+device-mapper-multipath-libs-0.8.7
 device-mapper-persistent-data-0.9.0
 dhcp-client-4.4.3
 dhcp-common-4.4.3
@@ -101,98 +103,106 @@
 dnf-utils-4.2.1
 dnsmasq-2.86
 dosfstools-4.2
-dracut-055
+dracut-056
 drpm-0.5.1
 dumb-init-1.2.5
 dwz-0.14
-e2fsprogs-1.46.3
-e2fsprogs-libs-1.46.3
+e2fsprogs-1.46.5
+e2fsprogs-libs-1.46.5
 ed-1.14.2
-edk2-ovmf-20211126gitbb1bba3d7767
+edk2-ovmf-20220526git16779ede2d36
 efi-filesystem-5
 efi-srpm-macros-5
-efivar-libs-37
-elfutils-0.186
-elfutils-debuginfod-client-0.186
-elfutils-default-yama-scope-0.186
-elfutils-libelf-0.186
-elfutils-libs-0.186
+efivar-libs-38
+elfutils-0.187
+elfutils-debuginfod-client-0.187
+elfutils-default-yama-scope-0.187
+elfutils-libelf-0.187
+elfutils-libs-0.187
 emacs-filesystem-27.2
-exempi-2.5.1
+exempi-2.6.0
 exiv2-0.27.5
 exiv2-libs-0.27.5
 expat-2.4.7
-fedora-gpg-keys-35
-fedora-messaging-3.0.0
-fedora-release-common-35
-fedora-release-container-35
-fedora-release-identity-container-35
-fedora-repos-35
-fedora-repos-modular-35
-fedora-repos-ostree-35
-file-5.40
-file-libs-5.40
-filesystem-3.14
-findutils-4.8.0
+fdk-aac-free-2.0.0
+fedora-gpg-keys-36
+fedora-messaging-3.0.2
+fedora-release-common-36
+fedora-release-container-36
+fedora-release-identity-container-36
+fedora-repos-36
+fedora-repos-modular-36
+fedora-repos-ostree-36
+file-5.41
+file-libs-5.41
+filesystem-3.16
+findutils-4.9.0
 flac-libs-1.3.4
+flatpak-1.12.7
+flatpak-selinux-1.12.7
+flatpak-session-helper-1.12.7
 fmt-8.1.1
-fontconfig-2.13.94
+fontconfig-2.14.0
 fonts-filesystem-2.0.5
 fonts-srpm-macros-2.0.5
 fpc-srpm-macros-1.3
-freetype-2.11.0
+freetype-2.12.1
 fribidi-1.0.11
 fuse-2.9.9
 fuse-common-3.10.5
 fuse-libs-2.9.9
-fuse-overlayfs-1.7.1
+fuse-overlayfs-1.8.1
 fuse3-3.10.5
 fuse3-libs-3.10.5
-gawk-5.1.0
-gc-8.0.4
-gcc-11.3.1
+gawk-5.1.1
+gc-8.0.6
+gcc-12.1.1
+gcc-plugin-annobin-12.1.1
 gdb-minimal-12.1
 gdbm-libs-1.22
 gdisk-1.0.9
-gdk-pixbuf2-2.42.6
-gdk-pixbuf2-modules-2.42.6
+gdk-pixbuf2-2.42.8
+gdk-pixbuf2-modules-2.42.8
 genisoimage-1.1.11
+geoclue2-2.6.0
 geolite2-city-20191217
 geolite2-country-20191217
 gettext-0.21
 gettext-libs-0.21
 ghc-srpm-macros-1.5.0
 giflib-5.2.1
-git-2.35.3
-git-core-2.35.3
-git-core-doc-2.35.3
-glib-networking-2.70.1
-glib2-2.70.5
-glibc-2.34
-glibc-common-2.34
-glibc-devel-2.34
-glibc-gconv-extra-2.34
-glibc-headers-x86-2.34
-glibc-minimal-langpack-2.34
-glibmm24-2.66.4
-glusterfs-client-xlators-9.5
-gmp-6.2.0
+git-2.36.1
+git-core-2.36.1
+git-core-doc-2.36.1
+glib-networking-2.72.0
+glib2-2.72.2
+glibc-2.35
+glibc-common-2.35
+glibc-devel-2.35
+glibc-gconv-extra-2.35
+glibc-headers-x86-2.35
+glibc-minimal-langpack-2.35
+glusterfs-client-xlators-10.2
+gmp-6.2.1
 gnat-srpm-macros-4
-gnupg2-2.3.4
+gnupg2-2.3.6
 gnutls-3.7.6
 gnutls-dane-3.7.6
 gnutls-utils-3.7.6
 go-srpm-macros-3.0.15
-gobject-introspection-1.70.0
-golang-1.16.15
-golang-bin-1.16.15
+gobject-introspection-1.72.0
+golang-1.18.3
+golang-bin-1.18.3
 golang-github-aliyun-cli-3.0.104
-golang-src-1.16.15
-gpg-pubkey-9867c58f
+golang-src-1.18.3
+google-noto-fonts-common-20201206
+google-noto-sans-vf-fonts-20201206
+gperftools-libs-2.9.1
+gpg-pubkey-38ab71f4
 gpgme-1.15.1
 graphene-1.10.6
 graphite2-1.3.14
-grep-3.6
+grep-3.7
 groff-1.22.4
 groff-base-1.22.4
 grub2-common-2.06
@@ -202,7 +212,7 @@
 grub2-tools-2.06
 grub2-tools-extra-2.06
 grub2-tools-minimal-2.06
-gsettings-desktop-schemas-41.0
+gsettings-desktop-schemas-42.0
 gsm-1.0.19
 gssproxy-0.8.4
 gstreamer1-1.20.0
@@ -211,26 +221,25 @@
 gtk3-3.24.34
 guestfs-tools-1.48.2
 guile22-2.2.7
-gzip-1.10
-harfbuzz-2.9.1
+gzip-1.11
+harfbuzz-4.0.0
 hexedit-1.6
 hicolor-icon-theme-0.17
 hivex-libs-1.3.21
 hwdata-0.360
 ignition-validate-2.14.0
-ima-evm-utils-1.3.2
+ima-evm-utils-1.4
 info-6.8
 inih-55
 ipcalc-1.0.1
-iproute-5.13.0
-iproute-tc-5.13.0
+iproute-5.15.0
+iproute-tc-5.15.0
 iptables-legacy-1.8.7
 iptables-legacy-libs-1.8.7
 iptables-libs-1.8.7
-iputils-20210722
+iputils-20211215
 ipxe-roms-qemu-20200823
-iso-codes-4.6.0
-jack-audio-connection-kit-1.9.19
+iso-codes-4.9.0
 jansson-2.13.1
 jbigkit-libs-2.1
 jq-1.6
@@ -238,9 +247,9 @@
 json-glib-1.6.6
 kbd-2.4.0
 kbd-misc-2.4.0
-kernel-core-5.18.5
-kernel-headers-5.18.4
-kernel-modules-5.18.5
+kernel-core-5.17.13
+kernel-headers-5.17.11
+kernel-modules-5.17.13
 kernel-srpm-macros-1.0
 keyutils-1.6.1
 keyutils-libs-1.6.1
@@ -248,12 +257,12 @@
 kmod-libs-29
 koji-1.29.0
 koji-utils-1.29.0
-kpartx-0.8.6
+kpartx-0.8.7
 krb5-libs-1.19.2
 krb5-pkinit-1.19.2
 krb5-workstation-1.19.2
 langpacks-core-font-en-3.0
-lcms2-2.12
+lcms2-2.13.1
 less-590
 libX11-1.7.3.1
 libX11-common-1.7.3.1
@@ -264,8 +273,8 @@
 libXdamage-1.1.5
 libXext-1.3.4
 libXfixes-6.0.0
-libXft-2.3.3
-libXi-1.7.10
+libXft-2.3.4
+libXi-1.8
 libXinerama-1.1.4
 libXrandr-1.5.2
 libXrender-0.9.10
@@ -274,29 +283,30 @@
 libXxf86vm-1.1.4
 libacl-2.3.1
 libaio-0.3.111
+libappstream-glib-0.7.18
 libarchive-3.5.3
 libargon2-20171227
 libassuan-2.5.5
 libasyncns-0.8
 libattr-2.5.1
 libbasicobjects-0.1.1
-libblkid-2.37.4
-libbpf-0.6.1
+libblkid-2.38
+libbpf-0.7.0
 libbrotli-1.0.9
 libbsd-0.10.0
-libcacard-2.8.0
+libcacard-2.8.1
 libcanberra-0.30
 libcanberra-gtk3-0.30
 libcap-2.48
-libcap-ng-0.8.2
+libcap-ng-0.8.3
 libcbor-0.7.0
 libcloudproviders-0.3.1
 libcollection-0.7.0
-libcom_err-1.46.3
+libcom_err-1.46.5
 libcomps-0.1.18
 libconfig-1.7.3
 libcue-2.2.1
-libcurl-7.79.1
+libcurl-7.82.0
 libdatrie-0.2.13
 libdb-5.3.28
 libdecor-0.1.0
@@ -304,131 +314,128 @@
 libdrm-2.4.110
 libeconf-0.4.0
 libedit-3.1
-libepoxy-1.5.9
+libepoxy-1.5.10
 libev-4.33
 libevent-2.1.12
-libexif-0.6.23
-libfdisk-2.37.4
+libexif-0.6.24
+libfdisk-2.38
 libfdt-1.6.1
-libffado-2.4.5
-libffi-3.1
-libfido2-1.8.0
+libffi-3.4.2
+libfido2-1.10.0
 libfsverity-1.4
-libgcc-11.3.1
-libgcrypt-1.9.4
+libgcc-12.1.1
+libgcrypt-1.10.1
 libgexiv2-0.14.0
-libgfapi0-9.5
-libgfrpc0-9.5
-libgfxdr0-9.5
-libglusterfs0-9.5
-libglvnd-1.3.4
-libglvnd-egl-1.3.4
-libglvnd-glx-1.3.4
-libgomp-11.3.1
-libgpg-error-1.43
+libgfapi0-10.2
+libgfrpc0-10.2
+libgfxdr0-10.2
+libglusterfs0-10.2
+libglvnd-1.4.0
+libglvnd-egl-1.4.0
+libglvnd-glx-1.4.0
+libgomp-12.1.1
+libgpg-error-1.45
 libgrss-0.7.0
 libgsf-1.14.47
 libgudev-237
 libguestfs-1.48.3
 libguestfs-appliance-1.48.3
 libguestfs-xfs-1.48.3
-libgusb-0.3.9
+libgusb-0.3.10
 libgxps-0.3.2
 libibverbs-39.0
 libicu-69.1
 libidn2-2.3.2
-libiec61883-1.2.0
-libimobiledevice-1.3.0
 libini_config-1.3.1
 libiptcdata-1.0.5
 libiscsi-1.19.0
-libjpeg-turbo-2.1.0
+libjpeg-turbo-2.1.2
 libkadm5-1.19.2
 libkcapi-1.3.1
 libkcapi-hmaccalc-1.3.1
 libksba-1.6.0
+libldac-2.0.2.3
 libmaxminddb-1.6.0
 libmnl-1.0.4
 libmodulemd-2.14.0
-libmount-2.37.4
+libmount-2.38
 libmpc-1.2.1
 libnet-1.2
 libnetfilter_conntrack-1.0.8
 libnfnetlink-1.0.1
 libnfs-4.0.0
-libnfsidmap-2.5.4
-libnftnl-1.2.0
-libnghttp2-1.45.1
-libnl3-3.5.0
-libnsl2-1.3.0
+libnfsidmap-2.6.1
+libnftnl-1.2.1
+libnghttp2-1.46.0
+libnl3-3.6.0
+libnotify-0.7.12
+libnsl2-2.0.0
 libogg-1.3.5
-libosinfo-1.9.0
+libosinfo-1.10.0
 libpath_utils-0.2.1
 libpcap-1.10.1
 libpciaccess-0.16
-libpipeline-1.5.3
+libpipeline-1.5.5
 libpkgconf-1.8.0
-libplist-2.2.0
-libpmem-1.11.0
+libpmem-1.11.1
 libpng-1.6.37
-libpq-13.4
+libpq-14.1
 libproxy-0.4.17
+libproxy-webkitgtk4-0.4.17
 libpsl-0.21.1
 libpwquality-1.4.4
 librados2-16.2.9
-libraw1394-2.1.2
 librbd1-16.2.9
 librdmacm-39.0
 libref_array-0.1.5
 librepo-1.14.3
-libreport-filesystem-2.15.2
+libreport-filesystem-2.17.1
 libretls-3.5.2
-libsamplerate-0.2.2
+librsvg2-2.54.3
+libsbc-1.5
 libseccomp-2.5.3
 libselinux-3.3
 libselinux-utils-3.3
 libsemanage-3.3
 libsepol-3.3
 libserf-1.3.9
-libsigc++20-2.10.7
-libsigsegv-2.13
+libsigsegv-2.14
 libslirp-4.6.1
-libsmartcols-2.37.4
+libsmartcols-2.38
 libsndfile-1.0.31
 libsolv-0.7.22
 libsoup-2.74.2
-libss-1.46.3
+libss-1.46.5
 libssh-0.9.6
 libssh-config-0.9.6
 libssh2-1.10.0
-libstdc++-11.3.1
+libstdc++-12.1.1
 libstemmer-0
-libtasn1-4.16.0
-libtdb-1.4.4
-libthai-0.1.28
+libtasn1-4.18.0
+libtdb-1.4.6
+libthai-0.1.29
 libtheora-1.1.1
-libtiff-4.4.0
+libtiff-4.3.0
 libtirpc-1.3.2
-libtool-ltdl-2.4.6
+libtool-ltdl-2.4.7
 libtpms-0.9.4
-libtracker-sparql-3.2.1
-libunistring-0.9.10
-libunwind-1.5.0
+libtracker-sparql-3.3.1
+libunistring-1.0
+libunwind-1.6.2
 liburing-2.0
 libusal-1.1.11
 libusb1-1.0.25
-libusbmuxd-2.0.2
 libutempter-1.2.1
-libuuid-2.37.4
+libuuid-2.38
 libverto-0.3.2
 libverto-libev-0.3.2
-libvirt-daemon-7.6.0
-libvirt-daemon-config-network-7.6.0
-libvirt-daemon-driver-network-7.6.0
-libvirt-daemon-driver-qemu-7.6.0
-libvirt-daemon-driver-secret-7.6.0
-libvirt-daemon-driver-storage-core-7.6.0
-libvirt-libs-7.6.0
+libvirt-daemon-8.1.0
+libvirt-daemon-config-network-8.1.0
+libvirt-daemon-driver-network-8.1.0
+libvirt-daemon-driver-qemu-8.1.0
+libvirt-daemon-driver-secret-8.1.0
+libvirt-daemon-driver-storage-core-8.1.0
+libvirt-libs-8.1.0
 libvisual-0.4.0
 libvorbis-1.3.7
 libwayland-client-1.20.0
@@ -436,23 +443,24 @@
 libwayland-egl-1.20.0
 libwayland-server-1.20.0
 libwebp-1.2.2
-libwsman1-2.7.0
+libwsman1-2.7.1
 libxcb-1.13.1
 libxcrypt-4.4.28
 libxcrypt-devel-4.4.28
-libxkbcommon-1.3.1
-libxml++-2.42.1
+libxkbcommon-1.4.0
 libxml2-2.9.14
 libxshmfence-1.3
-libxslt-1.1.34
+libxslt-1.1.35
 libyaml-0.2.5
 libzstd-1.5.2
+lilv-libs-0.24.12
 linux-atm-libs-2.5.1
 linux-firmware-20220509
 linux-firmware-whence-20220509
-llvm-libs-13.0.0
+llvm-libs-14.0.0
+low-memory-monitor-2.1
 lsscsi-0.32
-lttng-ust-2.12.2
+lttng-ust-2.13.1
 lua-libs-5.4.4
 lua-srpm-macros-1
 lvm2-2.03.11
@@ -461,19 +469,20 @@
 lzo-2.10
 lzop-1.04
 make-4.3
-man-db-2.9.4
+malcontent-libs-0.10.3
+man-db-2.10.0
 mdadm-4.2
 memstrack-0.2.4
-mercurial-5.9.3
-mesa-dri-drivers-21.3.8
-mesa-filesystem-21.3.8
-mesa-libEGL-21.3.8
-mesa-libGL-21.3.8
-mesa-libgbm-21.3.8
-mesa-libglapi-21.3.8
-mesa-vulkan-drivers-21.3.8
+mercurial-6.1.3
+mesa-dri-drivers-22.1.1
+mesa-filesystem-22.1.1
+mesa-libEGL-22.1.1
+mesa-libGL-22.1.1
+mesa-libgbm-22.1.1
+mesa-libglapi-22.1.1
+mesa-vulkan-drivers-22.1.1
 mokutil-0.6.0
-mozjs78-78.15.0
+mozjs91-91.9.0
 mpdecimal-2.5.1
 mpfr-4.1.0
 mtools-4.0.40
@@ -484,8 +493,8 @@
 netavark-1.0.3
 netcat-1.218
 nettle-3.7.3
-nfs-utils-2.5.4
-nftables-1.0.0
+nfs-utils-2.6.1
+nftables-1.0.1
 nim-srpm-macros-3
 npth-1.6
 nspr-4.34.0
@@ -504,23 +513,26 @@
 oniguruma-6.9.8
 openblas-srpm-macros-2
 openjpeg2-2.4.0
-openldap-2.4.59
-openssh-8.7p1
-openssh-clients-8.7p1
-openssl-1.1.1n
-openssl-libs-1.1.1n
-openssl-pkcs11-0.4.11
+openldap-2.6.2
+openldap-compat-2.6.2
+openssh-8.8p1
+openssh-clients-8.8p1
+openssl-3.0.3
+openssl-libs-3.0.3
 opus-1.3.1
 orc-0.4.31
 os-prober-1.77
 osinfo-db-20220516
-osinfo-db-tools-1.9.0
-ostree-2022.3
-ostree-libs-2022.3
-p11-kit-0.23.22
-p11-kit-trust-0.23.22
+osinfo-db-tools-1.10.0
+ostree-2022.4
+ostree-libs-2022.4
+p11-kit-0.24.1
+p11-kit-server-0.24.1
+p11-kit-trust-0.24.1
+package-notes-srpm-macros-0.4
 pam-1.5.2
-pango-1.50.4
+pam-libs-1.5.2
+pango-1.50.7
 paper-2.3
 parted-3.4
 patch-2.7.6
@@ -528,7 +540,7 @@
 pcre2-10.40
 pcre2-syntax-10.40
 pcre2-utf32-10.40
-pcsc-lite-libs-1.9.5
+pcsc-lite-libs-1.9.7
 perl-AutoLoader-5.74
 perl-B-1.82
 perl-Carp-1.52
@@ -538,7 +550,7 @@
 perl-Digest-1.20
 perl-Digest-MD5-2.58
 perl-DynaLoader-1.50
-perl-Encode-3.15
+perl-Encode-3.17
 perl-Errno-1.33
 perl-Error-0.17029
 perl-Exporter-5.76
@@ -554,12 +566,12 @@
 perl-FileHandle-2.03
 perl-Getopt-Long-2.52
 perl-Getopt-Std-1.13
-perl-Git-2.35.3
+perl-Git-2.36.1
 perl-HTTP-Tiny-0.080
 perl-I18N-Langinfo-0.19
 perl-IO-1.46
 perl-IO-Socket-IP-0.41
-perl-IO-Socket-SSL-2.072
+perl-IO-Socket-SSL-2.074
 perl-IPC-Open3-1.21
 perl-IPC-Run3-0.048
 perl-List-MoreUtils-0.430
@@ -567,7 +579,7 @@
 perl-MIME-Base64-3.16
 perl-Mozilla-CA-20211001
 perl-NDBM_File-1.15
-perl-Net-SSLeay-1.90
+perl-Net-SSLeay-1.92
 perl-POSIX-1.97
 perl-Params-Util-1.102
 perl-PathTools-3.80
@@ -577,17 +589,17 @@
 perl-Pod-Usage-2.01
 perl-Scalar-List-Utils-1.62
 perl-SelectSaver-1.02
-perl-Socket-2.032
+perl-Socket-2.033
 perl-Storable-3.25
 perl-Symbol-1.09
 perl-Sys-Guestfs-1.48.3
 perl-Term-ANSIColor-5.01
 perl-Term-Cap-1.17
 perl-TermReadKey-2.38
-perl-Text-ParseWords-3.30
+perl-Text-ParseWords-3.31
 perl-Text-Tabs+Wrap-2021.0814
 perl-Time-Local-1.300
-perl-URI-5.09
+perl-URI-5.10
 perl-base-2.27
 perl-constant-1.33
 perl-hivex-1.3.21
@@ -606,87 +618,94 @@
 perl-srpm-macros-1
 perl-subs-1.04
 perl-vars-1.05
-pigz-2.5
+pigz-2.7
+pipewire-0.3.52
+pipewire-alsa-0.3.52
+pipewire-jack-audio-connection-kit-0.3.52
+pipewire-libs-0.3.52
+pipewire-pulseaudio-0.3.52
 pixman-0.40.0
 pkgconf-1.8.0
 pkgconf-m4-1.8.0
 pkgconf-pkg-config-1.8.0
-podman-3.4.7
-podman-gvproxy-3.4.7
-podman-plugins-3.4.7
+podman-4.1.0
 policycoreutils-3.3
 policycoreutils-python-utils-3.3
 polkit-0.120
 polkit-libs-0.120
 polkit-pkla-compat-0.1
-poppler-21.08.0
-poppler-data-0.4.9
-poppler-glib-21.08.0
+poppler-22.01.0
+poppler-data-0.4.11
+poppler-glib-22.01.0
 popt-1.18
 procps-ng-3.3.17
 protobuf-c-1.4.0
 psmisc-23.4
-psutils-2.06
+psutils-2.07
 publicsuffix-list-dafsa-20210518
 pulseaudio-libs-15.0
-pylint-2.9.6
-python-pip-wheel-21.2.3
-python-setuptools-wheel-57.4.0
+pulseaudio-utils-15.0
+pylint-2.13.7
+python-pip-wheel-21.3.1
+python-setuptools-wheel-59.6.0
 python-srpm-macros-3.10
-python-unversioned-command-3.10.5
-python3-3.10.5
+python3-3.10.4
 python3-Automat-20.2.0
-python3-astroid-2.6.6
+python3-astroid-2.11.5
 python3-atpublic-1.0
-python3-attrs-21.2.0
+python3-attrs-21.4.0
 python3-audit-3.0.8
 python3-blinker-1.4
-python3-boto3-1.20.50
-python3-botocore-1.23.50
+python3-boto3-1.21.33
+python3-botocore-1.24.33
 python3-cffi-1.15.0
-python3-charset-normalizer-2.0.4
-python3-click-8.0.1
+python3-charset-normalizer-2.0.11
+python3-click-8.0.4
 python3-colorama-0.4.4
 python3-constantly-15.1.0
-python3-coverage+toml-5.6
-python3-coverage-5.6
+python3-coverage+toml-6.3
+python3-coverage-6.3
 python3-createrepo_c-0.20.0
 python3-crochet-1.10.0
-python3-cryptography-3.4.7
+python3-cryptography-36.0.0
 python3-dateutil-2.8.1
 python3-dbus-1.2.18
-python3-decorator-5.0.9
+python3-decorator-5.1.1
+python3-dill-0.3.4
 python3-distro-1.6.0
 python3-dnf-4.13.0
 python3-dnf-plugins-core-4.2.1
-python3-docutils-0.16
-python3-fedora-messaging-3.0.0
-python3-flake8-3.9.2
+python3-docutils-0.17.1
+python3-fb-re2-1.0.7
+python3-fedora-messaging-3.0.2
+python3-flake8-4.0.1
 python3-flufl-lock-3.2
-python3-gobject-base-3.42.0
+python3-gobject-base-3.42.1
+python3-gobject-base-noarch-3.42.1
 python3-gpg-1.15.1
-python3-gssapi-1.6.14
+python3-gssapi-1.7.2
 python3-hawkey-0.67.0
 python3-hyperlink-21.0.0
-python3-idna-3.2
+python3-idna-3.3
 python3-incremental-21.3.0
 python3-iniconfig-1.1.1
-python3-isort-5.9.3
-python3-jmespath-0.10.0
+python3-isort-5.10.1
+python3-jmespath-1.0.0
 python3-jsonschema-3.2.0
 python3-koji-1.29.0
 python3-koji-cli-plugins-1.29.0
-python3-lazy-object-proxy-1.6.0
+python3-lazy-object-proxy-1.7.1
 python3-libcomps-0.1.18
 python3-libdnf-0.67.0
-python3-libs-3.10.5
+python3-libs-3.10.4
 python3-libselinux-3.3
 python3-libsemanage-3.3
-python3-libvirt-7.6.0
+python3-libvirt-8.0.0
 python3-mccabe-0.6.1
 python3-packaging-21.3
 python3-pika-1.2.1
-python3-pluggy-0.13.1
+python3-platformdirs-2.3.0
+python3-pluggy-1.0.0
 python3-ply-3.11
 python3-policycoreutils-3.3
 python3-psycopg2-2.9.1
@@ -694,18 +713,17 @@
 python3-pyOpenSSL-21.0.0
 python3-pyasn1-0.4.8
 python3-pyasn1-modules-0.4.8
-python3-pycodestyle-2.7.0
+python3-pycodestyle-2.8.0
 python3-pycparser-2.20
-python3-pycurl-7.44.1
-python3-pyflakes-2.3.1
-python3-pylint-2.9.6
+python3-pyflakes-2.4.0
+python3-pylint-2.13.7
 python3-pyparsing-2.4.7
 python3-pyrsistent-0.18.1
 python3-pysocks-1.7.1
-python3-pytest-6.2.4
+python3-pytest-6.2.5
 python3-pytest-cov-3.0.0
 python3-pytz-2022.1
-python3-pyyaml-5.4.1
+python3-pyyaml-6.0
 python3-requests-2.27.1
 python3-requests-gssapi-1.2.3
 python3-rpm-4.17.0
@@ -714,147 +732,159 @@
 python3-semver-2.13.0
 python3-service-identity-21.1.0
 python3-setools-4.4.0
-python3-setuptools-57.4.0
+python3-setuptools-59.6.0
 python3-six-1.16.0
 python3-tenacity-8.0.1
 python3-toml-0.10.2
+python3-tomli-1.2.3
 python3-tornado-6.1.0
 python3-twisted+tls-21.7.0
 python3-twisted-21.7.0
-python3-typed_ast-1.4.3
+python3-typed_ast-1.5.4
 python3-typing-extensions-3.10.0.2
-python3-unbound-1.16.0
-python3-urllib3-1.26.7
-python3-wrapt-1.12.1
-python3-zope-event-4.2.0
+python3-urllib3-1.26.8
+python3-wrapt-1.14.0
 python3-zope-interface-5.4.0
-qemu-audio-alsa-6.1.0
-qemu-audio-jack-6.1.0
-qemu-audio-oss-6.1.0
-qemu-audio-pa-6.1.0
-qemu-audio-sdl-6.1.0
-qemu-audio-spice-6.1.0
-qemu-block-curl-6.1.0
-qemu-block-dmg-6.1.0
-qemu-block-gluster-6.1.0
-qemu-block-iscsi-6.1.0
-qemu-block-nfs-6.1.0
-qemu-block-rbd-6.1.0
-qemu-block-ssh-6.1.0
-qemu-char-baum-6.1.0
-qemu-char-spice-6.1.0
-qemu-common-6.1.0
-qemu-device-display-qxl-6.1.0
-qemu-device-display-vhost-user-gpu-6.1.0
-qemu-device-display-virtio-gpu-6.1.0
-qemu-device-display-virtio-gpu-gl-6.1.0
-qemu-device-display-virtio-gpu-pci-6.1.0
-qemu-device-display-virtio-gpu-pci-gl-6.1.0
-qemu-device-display-virtio-vga-6.1.0
-qemu-device-display-virtio-vga-gl-6.1.0
-qemu-device-usb-host-6.1.0
-qemu-device-usb-redirect-6.1.0
-qemu-device-usb-smartcard-6.1.0
-qemu-img-6.1.0
-qemu-kvm-6.1.0
-qemu-kvm-core-6.1.0
-qemu-system-x86-6.1.0
-qemu-system-x86-core-6.1.0
-qemu-ui-curses-6.1.0
-qemu-ui-egl-headless-6.1.0
-qemu-ui-gtk-6.1.0
-qemu-ui-opengl-6.1.0
-qemu-ui-sdl-6.1.0
-qemu-ui-spice-app-6.1.0
-qemu-ui-spice-core-6.1.0
+qemu-audio-alsa-6.2.0
+qemu-audio-jack-6.2.0
+qemu-audio-oss-6.2.0
+qemu-audio-pa-6.2.0
+qemu-audio-sdl-6.2.0
+qemu-audio-spice-6.2.0
+qemu-block-curl-6.2.0
+qemu-block-dmg-6.2.0
+qemu-block-gluster-6.2.0
+qemu-block-iscsi-6.2.0
+qemu-block-nfs-6.2.0
+qemu-block-rbd-6.2.0
+qemu-block-ssh-6.2.0
+qemu-char-baum-6.2.0
+qemu-char-spice-6.2.0
+qemu-common-6.2.0
+qemu-device-display-qxl-6.2.0
+qemu-device-display-vhost-user-gpu-6.2.0
+qemu-device-display-virtio-gpu-6.2.0
+qemu-device-display-virtio-gpu-gl-6.2.0
+qemu-device-display-virtio-gpu-pci-6.2.0
+qemu-device-display-virtio-gpu-pci-gl-6.2.0
+qemu-device-display-virtio-vga-6.2.0
+qemu-device-display-virtio-vga-gl-6.2.0
+qemu-device-usb-host-6.2.0
+qemu-device-usb-redirect-6.2.0
+qemu-device-usb-smartcard-6.2.0
+qemu-img-6.2.0
+qemu-kvm-6.2.0
+qemu-kvm-core-6.2.0
+qemu-pr-helper-6.2.0
+qemu-system-x86-6.2.0
+qemu-system-x86-core-6.2.0
+qemu-ui-curses-6.2.0
+qemu-ui-egl-headless-6.2.0
+qemu-ui-gtk-6.2.0
+qemu-ui-opengl-6.2.0
+qemu-ui-sdl-6.2.0
+qemu-ui-spice-app-6.2.0
+qemu-ui-spice-core-6.2.0
+qemu-virtiofsd-6.2.0
 qrencode-libs-4.1.1
-qt5-srpm-macros-5.15.2
+qt5-srpm-macros-5.15.3
 quota-4.06
 quota-nls-4.06
-radvd-2.19
+re2-20211101
 readline-8.1
-redhat-rpm-config-201
+redhat-rpm-config-221
 rootfiles-8.1
 rpcbind-1.2.6
 rpm-4.17.0
 rpm-build-4.17.0
 rpm-build-libs-4.17.0
 rpm-libs-4.17.0
-rpm-ostree-2022.9
-rpm-ostree-libs-2022.9
+rpm-ostree-2022.10
+rpm-ostree-libs-2022.10
 rpm-plugin-selinux-4.17.0
 rpm-sign-libs-4.17.0
 rpmautospec-rpm-macros-0.2.6
 rsync-3.2.4
+rtkit-0.11
 rust-srpm-macros-21
 scrub-2.6.1
-seabios-bin-1.15.0
-seavgabios-bin-1.15.0
+seabios-bin-1.16.0
+seavgabios-bin-1.16.0
 sed-4.8
-selinux-policy-35.17
-selinux-policy-targeted-35.17
+selinux-policy-36.10
+selinux-policy-targeted-36.10
+serd-0.30.10
 setup-2.13.10
 sgabios-bin-0.20180715git
-shadow-utils-4.9
-shadow-utils-subid-4.9
+shadow-utils-4.11.1
+shadow-utils-subid-4.11.1
 shared-mime-info-2.1
-shim-x64-15.6
-skopeo-1.7.0
-slirp4netns-1.1.12
+shim-x64-15.4
+skopeo-1.8.0
+slirp4netns-1.2.0
 snappy-1.1.9
+sord-0.16.8
 sound-theme-freedesktop-0.8
 spice-server-0.15.0
 sqlite-libs-3.36.0
 squashfs-tools-4.5.1
+sratom-0.6.8
 strace-5.16
 subversion-1.14.1
 subversion-libs-1.14.1
-sudo-1.9.7p2
+sudo-1.9.8
 supermin-5.3.2
-swtpm-0.7.2
-swtpm-libs-0.7.2
-swtpm-tools-0.7.2
+swtpm-0.7.3
+swtpm-libs-0.7.3
+swtpm-tools-0.7.3
 syslinux-6.04
 syslinux-extlinux-6.04
 syslinux-extlinux-nonlinux-6.04
 syslinux-nonlinux-6.04
-systemd-249.12
-systemd-container-249.12
-systemd-libs-249.12
-systemd-networkd-249.12
-systemd-pam-249.12
-systemd-resolved-249.12
-systemd-rpm-macros-249.12
-systemd-udev-249.12
+systemd-250.7
+systemd-container-250.7
+systemd-libs-250.7
+systemd-networkd-250.7
+systemd-pam-250.7
+systemd-resolved-250.7
+systemd-rpm-macros-250.7
+systemd-udev-250.7
 tar-1.34
 totem-pl-parser-3.26.6
-tpm2-tss-3.1.0
-tracker-3.2.1
-tracker-miners-3.2.2
+tpm2-tss-3.2.0
+tracker-3.3.1
+tracker-miners-3.3.1
 trousers-0.3.15
 trousers-lib-0.3.15
 tzdata-2022a
 uchardet-0.0.6
-unbound-libs-1.16.0
+unbound-libs-1.15.0
 unzip-6.0
-upower-0.99.13
+upower-0.99.19
 usbredir-0.12.0
 userspace-rcu-0.13.0
-utf8proc-2.6.1
-util-linux-2.37.4
-util-linux-core-2.37.4
-vim-data-8.2.5085
-vim-minimal-8.2.5085
+utf8proc-2.7.0
+util-linux-2.38
+util-linux-core-2.38
+vim-data-8.2.5052
+vim-minimal-8.2.5052
 virglrenderer-0.9.1
 virt-win-reg-1.48.2
-vte-profile-0.66.2
-vte291-0.66.2
+vte-profile-0.68.0
+vte291-0.68.0
 vulkan-loader-1.3.204.0
+webkit2gtk3-jsc-2.36.3
+webrtc-audio-processing-0.3.1
 which-2.21
-xen-libs-4.15.2
-xen-licenses-4.15.2
-xfsprogs-5.12.0
-xkeyboard-config-2.33
+wireplumber-0.4.10
+wireplumber-libs-0.4.10
+xdg-dbus-proxy-0.1.3
+xdg-desktop-portal-1.12.4
+xdg-desktop-portal-gtk-1.12.0
+xen-libs-4.16.1
+xen-licenses-4.16.1
+xfsprogs-5.14.2
+xkeyboard-config-2.35.1
 xml-common-0.6.3
 xxhash-libs-0.8.1
 xz-5.2.5

@miabbott
Copy link
Member Author

miabbott commented Jun 22, 2022

Using the F35-based cosa container has us using ostree-2022.3/rpm-ostree-2022.9 vs. ostree-2022.4/rpm-ostree-2022.10 in the F36 version

https://github.com/ostreedev/ostree/releases/tag/v2022.4

https://github.com/coreos/rpm-ostree/releases/tag/v2022.10

Nothing jumps out as being required for building RHCOS 4.11; looking for input from @jlebon and/or @cgwalters

@cgwalters
Copy link
Member

My gut says this is related to running F36 binaries in a container on the RHEL7 host where the downstream RHCOS pipelines are running in privileged PSI. (More specifically a RHEL8 guest via F36 qemu on a RHEL7 host)

Yeah, almost certainly. We could try downgrading to f35 qemu; iirc we did something like this in the past. (sudo yum --setopt=releasever=35 distro-sync qemu-kvm, will take us to 6.1.0 vs 6.2)

@miabbott
Copy link
Member Author

Yeah, almost certainly. We could try downgrading to f35 qemu; iirc we did something like this in the past. (sudo yum --setopt=releasever=35 distro-sync qemu-kvm, will take us to 6.1.0 vs 6.2)

Running into dependency hell here...using the distro-sync suggestion, I'm running into:

Error: Transaction test error:
  file /usr/libexec/virtiofsd from install of qemu-common-2:6.1.0-15.fc35.x86_64 conflicts with file from package qemu-virtiofsd-2:6.2.0-12.fc36.x86_64

If I try removing the conflicting package first, it pulls out an ungodly amount of packages and the distro-sync operation is not useful:

+ yum -y remove qemu-virtiofsd                                                                                                                                 
Repository 'f36-coreos-continuous' is missing name in configuration, using id.                                                                                 
Dependencies resolved.                                                                                                                                         
==================================================================================================
 Package                                 Arch    Version                           Repo       Size
==================================================================================================                                                             
Removing:                                                                                                                                                      
 qemu-virtiofsd                          x86_64  2:6.2.0-12.fc36                   @updates  273 k
Removing dependent packages:                                                                                                                                   
 qemu-kvm                                x86_64  2:6.2.0-12.fc36                   @updates    0                                                               
Removing unused dependencies:                                                                                                                                  
 ModemManager-glib                       x86_64  1.18.8-1.fc36                     @updates  1.3 M
 NetworkManager-libnm                    x86_64  1:1.38.0-2.fc36                   @updates  9.3 M
 SDL2                                    x86_64  2.0.22-2.fc36                     @updates  1.7 M
 SDL2_image                              x86_64  2.0.5-8.fc36                      @fedora   165 k
 abattis-cantarell-fonts                 noarch  0.301-7.fc36                      @fedora   525 k
 adobe-source-code-pro-fonts             noarch  2.030.1.050-12.fc36               @fedora   1.8 M
 adwaita-cursor-theme                    noarch  42.0-1.fc36                       @fedora    12 M
 adwaita-icon-theme                      noarch  42.0-1.fc36                       @fedora   4.8 M                                                             
 alsa-lib                                x86_64  1.2.7.1-1.fc36                    @updates  1.4 M                                                             
 at-spi2-atk                             x86_64  2.38.0-4.fc36                     @fedora   273 k                                                             
 at-spi2-core                            x86_64  2.44.1-1.fc36                     @updates  518 k
 atk                                     x86_64  2.38.0-1.fc36                     @fedora   1.3 M
 avahi-glib                              x86_64  0.8-15.fc36                       @fedora    20 k
 avahi-libs                              x86_64  0.8-15.fc36                       @fedora   175 k
 bluez-libs                              x86_64  5.64-1.fc36                       @fedora   211 k
 boost-iostreams                         x86_64  1.76.0-11.fc36                    @updates   91 k
 boost-system                            x86_64  1.76.0-11.fc36                    @updates   17 k
 boost-thread                            x86_64  1.76.0-11.fc36                    @updates  121 k
 brlapi                                  x86_64  0.8.3-6.fc36                      @fedora   555 k
 brltty                                  x86_64  6.4-6.fc36                        @fedora   7.4 M
 cairo                                   x86_64  1.17.6-1.fc36                     @fedora   1.7 M
 cairo-gobject                           x86_64  1.17.6-1.fc36                     @fedora    43 k
 cdparanoia-libs                         x86_64  10.2-39.fc36                      @fedora   114 k
 colord-libs                             x86_64  1.4.6-1.fc36                      @fedora   849 k
 cups-libs                               x86_64  1:2.4.2-1.fc36                    @updates  672 k
 dconf                                   x86_64  0.40.0-6.fc36                     @fedora   305 k
 device-mapper-multipath-libs            x86_64  0.8.7-8.fc36                      @fedora   885 k
 exempi                                  x86_64  2.6.0-0.2.20211007gite23c213.fc36 @fedora   1.4 M
 exiv2                                   x86_64  0.27.5-2.fc36                     @fedora   4.4 M
 exiv2-libs                              x86_64  0.27.5-2.fc36                     @fedora   2.6 M
 fdk-aac-free                            x86_64  2.0.0-8.fc36                      @fedora   600 k
 flac-libs                               x86_64  1.3.4-1.fc36                      @fedora   555 k
 flatpak                                 x86_64  1.12.7-2.fc36                     @fedora   5.4 M
 flatpak-selinux                         noarch  1.12.7-2.fc36                     @fedora    12 k
 flatpak-session-helper                  x86_64  1.12.7-2.fc36                     @fedora   104 k
 fmt                                     x86_64  8.1.1-5.fc36                      @fedora   337 k
 fontconfig                              x86_64  2.14.0-1.fc36                     @fedora   830 k
 fonts-filesystem                        noarch  1:2.0.5-7.fc36                    @fedora     0  
 fribidi                                 x86_64  1.0.11-3.fc36                     @updates  198 k
 gdk-pixbuf2                             x86_64  2.42.8-1.fc36                     @fedora   2.5 M
 gdk-pixbuf2-modules                     x86_64  2.42.8-1.fc36                     @fedora   268 k
 geoclue2                                x86_64  2.6.0-2.fc36                      @fedora   461 k
 giflib                                  x86_64  5.2.1-11.fc36                     @fedora   114 k
 glib-networking                         x86_64  2.72.0-1.fc36                     @fedora   713 k
 glusterfs-client-xlators                x86_64  10.2-1.fc36                       @updates  3.7 M
 google-noto-fonts-common                noarch  20201206-9.fc36                   @fedora    15 k
 google-noto-sans-vf-fonts               noarch  20201206-9.fc36                   @fedora   1.1 M
 gperftools-libs                         x86_64  2.9.1-3.fc36                      @fedora   1.4 M
 graphene                                x86_64  1.10.6-4.fc36                     @fedora   167 k
 gsettings-desktop-schemas               x86_64  42.0-1.fc36                       @fedora   4.6 M
 gsm                                     x86_64  1.0.19-7.fc36                     @fedora    68 k
 gstreamer1                              x86_64  1.20.0-1.fc36                     @fedora   4.7 M
 gstreamer1-plugins-base                 x86_64  1.20.0-1.fc36                     @fedora   7.0 M
 gtk-update-icon-cache                   x86_64  3.24.34-1.fc36                    @updates   67 k
 gtk3                                    x86_64  3.24.34-1.fc36                    @updates   20 M
 hicolor-icon-theme                      noarch  0.17-13.fc36                      @updates   72 k
 hwdata                                  noarch  0.360-1.fc36                      @updates  8.4 M
 iso-codes                               noarch  4.9.0-2.fc36                      @fedora    18 M
 jbigkit-libs                            x86_64  2.1-23.fc36                       @fedora   114 k
 langpacks-core-font-en                  noarch  3.0-23.fc36                       @updates  351  
 lcms2                                   x86_64  2.13.1-1.fc36                     @fedora   420 k
 libX11                                  x86_64  1.7.3.1-2.fc36                    @fedora   1.3 M
 libX11-common                           noarch  1.7.3.1-2.fc36                    @fedora   1.3 M
 libX11-xcb                              x86_64  1.7.3.1-2.fc36                    @fedora    15 k
 libXau                                  x86_64  1.0.9-8.fc36                      @fedora    64 k
 libXcomposite                           x86_64  0.4.5-7.fc36                      @fedora    42 k
 libXcursor                              x86_64  1.2.0-7.fc36                      @fedora    50 k
 libXdamage                              x86_64  1.1.5-7.fc36                      @fedora    36 k
 libXext                                 x86_64  1.3.4-8.fc36                      @fedora    94 k
 libXfixes                               x86_64  6.0.0-3.fc36                      @fedora    35 k
 libXft                                  x86_64  2.3.4-2.fc36                      @fedora   135 k
 libXi                                   x86_64  1.8-2.fc36                        @fedora    81 k
 libXinerama                             x86_64  1.1.4-10.fc36                     @fedora    19 k
 libXrandr                               x86_64  1.5.2-8.fc36                      @fedora    52 k
 libXrender                              x86_64  0.9.10-16.fc36                    @fedora    50 k
 libXtst                                 x86_64  1.2.3-16.fc36                     @fedora    38 k
 libXv                                   x86_64  1.0.11-16.fc36                    @fedora    30 k
 libXxf86vm                              x86_64  1.1.4-18.fc36                     @fedora    26 k
 libappstream-glib                       x86_64  0.7.18-4.fc36                     @fedora   1.4 M
 libasyncns                              x86_64  0.8-22.fc36                       @fedora    59 k
 libcacard                               x86_64  3:2.8.1-2.fc36                    @fedora   120 k
 libcanberra                             x86_64  0.30-28.fc36                      @fedora   280 k
 libcanberra-gtk3                        x86_64  0.30-28.fc36                      @fedora    75 k
 libcloudproviders                       x86_64  0.3.1-5.fc36                      @fedora   124 k
 libcue                                  x86_64  2.2.1-9.fc36                      @fedora    82 k
 libdatrie                               x86_64  0.2.13-3.fc36                     @fedora    58 k
 libdecor                                x86_64  0.1.0-2.fc36                      @fedora    97 k
 libdrm                                  x86_64  2.4.110-1.fc36                    @fedora   417 k
 libepoxy                                x86_64  1.5.10-1.fc36                     @fedora   1.2 M
 libexif                                 x86_64  0.6.24-2.fc36                     @fedora   2.5 M
 libgexiv2                               x86_64  0.14.0-2.fc36                     @fedora   260 k
 libgfapi0                               x86_64  10.2-1.fc36                       @updates  223 k
 libgfrpc0                               x86_64  10.2-1.fc36                       @updates  242 k
 libgfxdr0                               x86_64  10.2-1.fc36                       @updates   81 k
 libglusterfs0                           x86_64  10.2-1.fc36                       @updates  956 k
 libglvnd                                x86_64  1:1.4.0-2.fc36                    @fedora   783 k
 libglvnd-egl                            x86_64  1:1.4.0-2.fc36                    @fedora    74 k
 libglvnd-glx                            x86_64  1:1.4.0-2.fc36                    @fedora   679 k
 libgrss                                 x86_64  0.7.0-14.fc36                     @fedora   171 k
 libgsf                                  x86_64  1.14.47-5.fc36                    @fedora   941 k
 libgudev                                x86_64  237-2.fc36                        @fedora    88 k
 libgusb                                 x86_64  0.3.10-2.fc36                     @fedora   128 k
 libgxps                                 x86_64  0.3.2-3.fc36                      @fedora   193 k
 libiptcdata                             x86_64  1.0.5-10.fc36                     @fedora   173 k
 libiscsi                                x86_64  1.19.0-5.fc36                     @fedora   223 k
 libldac                                 x86_64  2.0.2.3-10.fc36                   @fedora    79 k
 libnfs                                  x86_64  4.0.0-6.fc36                      @fedora   431 k
 libnotify                               x86_64  0.7.12-1.fc36                     @updates  111 k
 libogg                                  x86_64  2:1.3.5-3.fc36                    @fedora    50 k
 libosinfo                               x86_64  1.10.0-1.fc36                     @fedora   1.2 M
 libpciaccess                            x86_64  0.16-6.fc36                       @fedora    49 k
 libproxy                                x86_64  0.4.17-4.fc36                     @fedora   154 k
 libproxy-webkitgtk4                     x86_64  0.4.17-4.fc36                     @fedora    32 k
 librados2                               x86_64  2:16.2.9-1.fc36                   @updates   13 M
 librbd1                                 x86_64  2:16.2.9-1.fc36                   @updates   14 M
 librsvg2                                x86_64  2.54.4-1.fc36                     @updates   11 M
 libsbc                                  x86_64  1.5-3.fc36                        @fedora    90 k
 libsndfile                              x86_64  1.0.31-7.fc36                     @fedora   527 k
 libsoup                                 x86_64  2.74.2-2.fc36                     @fedora   1.2 M
 libstemmer                              x86_64  0-18.585svn.fc36                  @fedora   344 k
 libtdb                                  x86_64  1.4.7-1.fc36                      @updates   97 k
 libthai                                 x86_64  0.1.29-2.fc36                     @fedora   780 k
 libtheora                               x86_64  1:1.1.1-31.fc36                   @fedora   471 k
 libtiff                                 x86_64  4.4.0-1.fc36                      @updates  565 k
 libtracker-sparql                       x86_64  3.3.1-1.fc36                      @updates  1.2 M
 libunwind                               x86_64  1.6.2-2.fc36                      @fedora   162 k
 libusb1                                 x86_64  1.0.25-8.fc36                     @fedora   165 k
 libvisual                               x86_64  1:0.4.0-35.fc36                   @fedora   423 k
 libvorbis                               x86_64  1:1.3.7-5.fc36                    @fedora   911 k
 libwayland-client                       x86_64  1.20.0-4.fc36                     @fedora    70 k
 libwayland-cursor                       x86_64  1.20.0-4.fc36                     @fedora    37 k
 libwayland-egl                          x86_64  1.20.0-4.fc36                     @fedora    17 k
 libwayland-server                       x86_64  1.20.0-4.fc36                     @fedora    87 k
 libwebp                                 x86_64  1.2.2-4.fc36                      @fedora   866 k
 libxcb                                  x86_64  1.13.1-9.fc36                     @fedora   1.1 M
 libxshmfence                            x86_64  1.3-10.fc36                       @fedora    16 k
 libxslt                                 x86_64  1.1.35-2.fc36                     @updates  762 k
 lilv-libs                               x86_64  0.24.12-5.fc36                    @fedora   126 k
 llvm-libs                               x86_64  14.0.0-1.fc36                     @fedora   104 M
 low-memory-monitor                      x86_64  2.1-5.fc36                        @fedora    71 k
 lttng-ust                               x86_64  2.13.1-2.fc36                     @fedora   1.2 M
 malcontent-libs                         x86_64  0.10.3-2.fc36                     @fedora   114 k
 mesa-dri-drivers                        x86_64  22.1.2-1.fc36                     @updates   82 M
 mesa-filesystem                         x86_64  22.1.2-1.fc36                     @updates  3.6 k
 mesa-libEGL                             x86_64  22.1.2-1.fc36                     @updates  279 k
 mesa-libGL                              x86_64  22.1.2-1.fc36                     @updates  470 k
 mesa-libgbm                             x86_64  22.1.2-1.fc36                     @updates   66 k
 mesa-libglapi                           x86_64  22.1.2-1.fc36                     @updates  238 k
 mesa-vulkan-drivers                     x86_64  22.1.2-1.fc36                     @updates   21 M
 nspr                                    x86_64  4.34.0-1.fc36                     @updates  322 k
 nss                                     x86_64  3.79.0-1.fc36                     @updates  1.9 M
 nss-softokn                             x86_64  3.79.0-1.fc36                     @updates  1.8 M
 nss-softokn-freebl                      x86_64  3.79.0-1.fc36                     @updates  745 k
 nss-sysinit                             x86_64  3.79.0-1.fc36                     @updates   19 k
 nss-util                                x86_64  3.79.0-1.fc36                     @updates  222 k
 openjpeg2                               x86_64  2.4.0-10.fc36                     @updates  381 k
 opus                                    x86_64  1.3.1-10.fc36                     @fedora   367 k
 orc                                     x86_64  0.4.31-7.fc36                     @fedora   603 k
 osinfo-db                               noarch  20220516-1.fc36                   @updates  2.8 M
 osinfo-db-tools                         x86_64  1.10.0-1.fc36                     @fedora   186 k
 p11-kit-server                          x86_64  0.24.1-2.fc36                     @fedora   1.0 M
 pango                                   x86_64  1.50.7-1.fc36                     @updates  995 k
 pcre2-utf32                             x86_64  10.40-1.fc36                      @updates  538 k
 pcsc-lite-libs                          x86_64  1.9.8-1.fc36                      @updates   45 k
 pipewire                                x86_64  0.3.52-3.fc36                     @updates   76 k
 pipewire-alsa                           x86_64  0.3.52-3.fc36                     @updates  159 k
 pipewire-jack-audio-connection-kit      x86_64  0.3.52-3.fc36                     @updates  506 k
 pipewire-libs                           x86_64  0.3.52-3.fc36                     @updates  6.7 M
 pipewire-pulseaudio                     x86_64  0.3.52-3.fc36                     @updates   23 k
 poppler                                 x86_64  22.01.0-3.fc36                    @fedora   3.7 M
 poppler-data                            noarch  0.4.11-2.fc36                     @fedora    12 M
 poppler-glib                            x86_64  22.01.0-3.fc36                    @fedora   506 k
 pulseaudio-libs                         x86_64  15.0-5.fc36                       @fedora   3.2 M
 pulseaudio-utils                        x86_64  15.0-5.fc36                       @fedora   203 k
 qemu-audio-alsa                         x86_64  2:6.2.0-12.fc36                   @updates   29 k
 qemu-audio-jack                         x86_64  2:6.2.0-12.fc36                   @updates   25 k
 qemu-audio-oss                          x86_64  2:6.2.0-12.fc36                   @updates   21 k
 qemu-audio-pa                           x86_64  2:6.2.0-12.fc36                   @updates   29 k
 qemu-audio-sdl                          x86_64  2:6.2.0-12.fc36                   @updates   25 k
 qemu-audio-spice                        x86_64  2:6.2.0-12.fc36                   @updates   20 k
 qemu-block-curl                         x86_64  2:6.2.0-12.fc36                   @updates   37 k
 qemu-block-dmg                          x86_64  2:6.2.0-12.fc36                   @updates   16 k
 qemu-block-gluster                      x86_64  2:6.2.0-12.fc36                   @updates   41 k
 qemu-block-iscsi                        x86_64  2:6.2.0-12.fc36                   @updates   55 k
 qemu-block-nfs                          x86_64  2:6.2.0-12.fc36                   @updates   38 k
 qemu-block-rbd                          x86_64  2:6.2.0-12.fc36                   @updates   42 k
 qemu-block-ssh                          x86_64  2:6.2.0-12.fc36                   @updates   39 k
 qemu-char-baum                          x86_64  2:6.2.0-12.fc36                   @updates   20 k
 qemu-char-spice                         x86_64  2:6.2.0-12.fc36                   @updates   21 k
 qemu-device-display-qxl                 x86_64  2:6.2.0-12.fc36                   @updates   92 k
 qemu-device-display-vhost-user-gpu      x86_64  2:6.2.0-12.fc36                   @updates  263 k
 qemu-device-display-virtio-gpu          x86_64  2:6.2.0-12.fc36                   @updates   65 k
 qemu-device-display-virtio-gpu-gl       x86_64  2:6.2.0-12.fc36                   @updates   39 k
 qemu-device-display-virtio-gpu-pci      x86_64  2:6.2.0-12.fc36                   @updates   16 k
 qemu-device-display-virtio-gpu-pci-gl   x86_64  2:6.2.0-12.fc36                   @updates   16 k
 qemu-device-display-virtio-vga          x86_64  2:6.2.0-12.fc36                   @updates   17 k
 qemu-device-display-virtio-vga-gl       x86_64  2:6.2.0-12.fc36                   @updates   16 k
 qemu-device-usb-host                    x86_64  2:6.2.0-12.fc36                   @updates   53 k
 qemu-device-usb-redirect                x86_64  2:6.2.0-12.fc36                   @updates   70 k
 qemu-device-usb-smartcard               x86_64  2:6.2.0-12.fc36                   @updates   34 k
 qemu-pr-helper                          x86_64  2:6.2.0-12.fc36                   @updates  917 k
 qemu-system-x86                         x86_64  2:6.2.0-12.fc36                   @updates    0  
 qemu-ui-curses                          x86_64  2:6.2.0-12.fc36                   @updates   49 k
 qemu-ui-egl-headless                    x86_64  2:6.2.0-12.fc36                   @updates   16 k
 qemu-ui-gtk                             x86_64  2:6.2.0-12.fc36                   @updates   87 k
 qemu-ui-opengl                          x86_64  2:6.2.0-12.fc36                   @updates   33 k
 qemu-ui-sdl                             x86_64  2:6.2.0-12.fc36                   @updates   46 k
 qemu-ui-spice-app                       x86_64  2:6.2.0-12.fc36                   @updates   16 k
 qemu-ui-spice-core                      x86_64  2:6.2.0-12.fc36                   @updates   69 k
 rtkit                                   x86_64  0.11-30.fc36                      @updates  147 k
 serd                                    x86_64  0.30.10-4.fc36                    @fedora   139 k
 shared-mime-info                        x86_64  2.1-3.fc35                        @fedora   4.9 M
 sord                                    x86_64  0.16.8-3.fc36                     @fedora   109 k
 sound-theme-freedesktop                 noarch  0.8-17.fc36                       @fedora   460 k
 spice-server                            x86_64  0.15.0-4.fc36                     @fedora   1.2 M
 sratom                                  x86_64  0.6.8-2.fc36                      @fedora    47 k
 totem-pl-parser                         x86_64  3.26.6-4.fc36                     @fedora   314 k
 tracker                                 x86_64  3.3.1-1.fc36                      @updates  1.9 M
 tracker-miners                          x86_64  3.3.1-1.fc36                      @updates  4.1 M
 uchardet                                x86_64  0.0.6-15.fc36                     @fedora   227 k
 upower                                  x86_64  0.99.19-1.fc36                    @updates  390 k
 usbredir                                x86_64  0.12.0-2.fc36                     @fedora   115 k
 virglrenderer                           x86_64  0.9.1-3.20210420git36391559.fc36  @fedora   467 k
 vte-profile                             x86_64  0.68.0-1.fc36                     @fedora    53 k
 vte291                                  x86_64  0.68.0-1.fc36                     @fedora   775 k
 vulkan-loader                           x86_64  1.3.204.0-1.fc36                  @fedora   480 k
 webkit2gtk3-jsc                         x86_64  2.36.3-1.fc36                     @updates   20 M
 webrtc-audio-processing                 x86_64  0.3.1-8.fc36                      @fedora   726 k
 wireplumber                             x86_64  0.4.10-1.fc36                     @updates  280 k
 wireplumber-libs                        x86_64  0.4.10-1.fc36                     @updates  1.1 M
 xdg-dbus-proxy                          x86_64  0.1.3-1.fc36                      @fedora    90 k
 xdg-desktop-portal                      x86_64  1.12.4-1.fc36                     @fedora   1.8 M
 xdg-desktop-portal-gtk                  x86_64  1.12.0-5.fc36                     @fedora   483 k
 xml-common                              noarch  0.6.3-58.fc36                     @fedora    78 k

Transaction Summary
==================================================================================================
Remove  236 Packages

...

+ yum -y --setopt=releasever=35 --allowerasing distro-sync qemu-kvm
Repository 'f36-coreos-continuous' is missing name in configuration, using id.
Fedora 35 - x86_64                               21 MB/s |  79 MB     00:03     
Fedora 35 openh264 (From Cisco) - x86_64        4.5 kB/s | 2.5 kB     00:00     
Fedora Modular 35 - x86_64                      2.1 MB/s | 3.3 MB     00:01     
Fedora 35 - x86_64 - Updates                     18 MB/s |  30 MB     00:01     
Fedora Modular 35 - x86_64 - Updates            4.2 MB/s | 3.2 MB     00:00     
Dependencies resolved.
Nothing to do.                        
Complete! 

My yum-fu is lacking here; I might just switch the internal cosa BuildConfig to use F35 to get us unblocked.

Additional suggestions welcome.

@cgwalters
Copy link
Member

One data point here is that building+testing FCOS on a rhel7 host seems to work. I suspect something got fixed in newer kernels here.

@cgwalters
Copy link
Member

My yum-fu is lacking here; I might just switch the internal cosa BuildConfig to use F35 to get us unblocked.

We can change the public Prow one in openshift/release for the 4.11 branch, and no need to make it private right?

That said...hmm, an interesting data point here is that hacking things to just go to f35 qemu doesn't fix it. So there's something else going on here.

cgwalters added a commit to cgwalters/coreos-assembler that referenced this issue Jun 24, 2022
Something changed here to cause older host kernels (e.g. rhel8)
to break but only when emulated on an even older (rhel7) host.

See openshift/os#862

Will try to file a proper bug soon, but for now...
@cgwalters
Copy link
Member

Ah hah! 🕵️ I found it; sudo yum --setopt=releasever=35 distro-sync edk2-ovmf fixes things here. PR in coreos/coreos-assembler#2943

@cgwalters
Copy link
Member

(What helped me figure this out was that the problem was only occuring in uefi-secure, which was a tip off that it was somehow related to UEFI, and edk2-ovmf is the UEFI implementation qemu loads)

@miabbott
Copy link
Member Author

miabbott commented Jun 24, 2022

We can change the public Prow one in openshift/release for the 4.11 branch, and no need to make it private right?

I was gun-shy about changing the more broadly used cosa image since it would be using an older version of ostree/rpm-ostree and may interrupt other development work (e.g. CoreOS Layering)

@cgwalters
Copy link
Member

Switching the 4.11 branch won't stop layering work which is mainly now for 4.12, also I think we really do want to try to be consistent with what's used in Prow CI vs rebuilds.

@miabbott
Copy link
Member Author

Ah, I read your comment too quickly and didn't catch the mention of the 4.11 branch; definitely would make sense in that arrangement.

jlebon pushed a commit to coreos/coreos-assembler that referenced this issue Jun 24, 2022
Something changed here to cause older host kernels (e.g. rhel8)
to break but only when emulated on an even older (rhel7) host.

See openshift/os#862

Will try to file a proper bug soon, but for now...
@miabbott
Copy link
Member Author

Looking at the two edk2-ovmf versions...

F36 = edk2-ovmf-20220526git16779ede2d36-3.fc36.noarch
F35 = edk2-ovmf-20211126gitbb1bba3d7767-1.fc35.noarch

tianocore/edk2@edk2-stable202111...edk2-stable202205

611 commits between the upstream stable tags 😢

@nikita-dubrovskii
Copy link
Contributor

this package is not available for others, except x86

coreos/coreos-assembler#2947

@cgwalters
Copy link
Member

This is fixed, also filed https://bugzilla.redhat.com/show_bug.cgi?id=2101573

HuijingHei pushed a commit to HuijingHei/coreos-assembler that referenced this issue Jun 29, 2022
Something changed here to cause older host kernels (e.g. rhel8)
to break but only when emulated on an even older (rhel7) host.

See openshift/os#862

Will try to file a proper bug soon, but for now...
HuijingHei pushed a commit to HuijingHei/coreos-assembler that referenced this issue Jun 29, 2022
Something changed here to cause older host kernels (e.g. rhel8)
to break but only when emulated on an even older (rhel7) host.

See openshift/os#862

Will try to file a proper bug soon, but for now...

(cherry picked from commit 31a7416)
cgwalters added a commit to coreos/coreos-assembler that referenced this issue Jun 29, 2022
Something changed here to cause older host kernels (e.g. rhel8)
to break but only when emulated on an even older (rhel7) host.

See openshift/os#862

Will try to file a proper bug soon, but for now...

(cherry picked from commit 31a7416)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants