dracut generates incorrect initramfs after dnf upgrade; broken boot

[zacharyfreeman70]

Describe the bug
Unable to boot from a software RAID after performing a dnf upgrade.

Distribution used
Fedora 34 (minimal install, see below for reproducible Kickstart file).

Dracut version

# rpm -qa | grep "dracut"
dracut-055-3.fc34.x86_64
dracut-config-rescue-055-3.fc34.x86_64

Init system

# rpm -qa | grep "systemd" | sort
rpm-plugin-systemd-inhibit-4.16.1.3-1.fc34.x86_64
systemd-248.7-1.fc34.x86_64
systemd-libs-248.7-1.fc34.x86_64
systemd-networkd-248.7-1.fc34.x86_64
systemd-oomd-defaults-248.7-1.fc34.x86_64
systemd-pam-248.7-1.fc34.x86_64
systemd-rpm-macros-248.7-1.fc34.noarch
systemd-udev-248.7-1.fc34.x86_64

To Reproduce

1. Install a fresh copy of Fedora 34. See below for minimal Kickstart file to assist with reproducing. After Anaconda finishes, you'll have a fresh Fedora 34 system, with kernel 5.11, running on 3 software RAID-1 arrays: /, /boot, and swap.
2. Run dnf upgrade dracut systemd to get the latest version of dracut and systemd immediately after installation.
3. Shutdown the system and unplug a drive from the RAID-1.
4. Power the system back on. Despite the missing disk, the system will still successfully boot and bring you to a login prompt.
5. Shutdown the system and plug the unplugged drive back in.
6. Power the system back on. Reassemble the RAID-1 if necessary.
7. Run dnf upgrade. dnf will install a new kernel, 5.13. dracut will automatically create a new 5.13 initramfs for you.
8. Shutdown the system and unplug a hard drive from the RAID-1, just like in Step 3.
9. Power the system back on. The system will NOT boot and you'll get dropped into the emergency shell.

Expected behavior
Be able to boot with a missing RAID-1 member after running dnf upgrade, just like I was able to after a fresh install of Fedora 34 in Steps 1-4.

Additional context

1. Machine is using a legacy/traditional BIOS (no UEFI).

2. Working Kickstart file for Minimal Fedora 34 install on top of software RAID-1.

3. Kernel parameters before/after dnf upgrade (1 difference in the vmlinuz version number of the BOOT_IMAGE argument, which is expected after installing a new kernel):

# cat /proc/cmdline #<-BEFORE DNF UPGRADE
BOOT_IMAGE=(mduuid/48134451083e8a98a3b3ff946c7ef07c)/vmlinuz-5.11.12-300.fc34.x86_64 root=UUID=a7b1b7a1-f74c-46db-aee7-11662ed52529 ro resume=UUID=3dac8ce5-7b99-4a91-aecf-39bff7199860 rd.md.uuid=9ef8fef0:b3b04ded:94a9f448:2ac6f557 rd.md.uuid=48134451:083e8a98:a3b3ff94:6c7ef07c rd.md.uuid=07cf2047:1d9f15c2:7e3cd6c5:ceb1e8fd rhgb quiet

# cat /proc/cmdline #<-AFTER DNF UPGRADE
BOOT_IMAGE=(mduuid/48134451083e8a98a3b3ff946c7ef07c)/vmlinuz-5.13.12-200.fc34.x86_64 root=UUID=a7b1b7a1-f74c-46db-aee7-11662ed52529 ro resume=UUID=3dac8ce5-7b99-4a91-aecf-39bff7199860 rd.md.uuid=9ef8fef0:b3b04ded:94a9f448:2ac6f557 rd.md.uuid=48134451:083e8a98:a3b3ff94:6c7ef07c rd.md.uuid=07cf2047:1d9f15c2:7e3cd6c5:ceb1e8fd rhgb quiet

4. GRUB2 defaults before/after dnf upgrade (no difference):

# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=UUID=3dac8ce5-7b99-4a91-aecf-39bff7199860 rd.md.uuid=9ef8fef0:b3b04ded:94a9f448:2ac6f557 rd.md.uuid=48134451:083e8a98:a3b3ff94:6c7ef07c rd.md.uuid=07cf2047:1d9f15c2:7e3cd6c5:ceb1e8fd rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

5. /etc/mdadm.conf before/after dnf upgrade (no difference):

# cat /etc/mdadm.conf
# mdadm.conf written out by anaconda
MAILADDR root
AUTO +imsm +1.x -all
ARRAY /dev/md/boot level=raid1 num-devices=2 UUID=48134451:083e8a98:a3b3ff94:6c7ef07c
ARRAY /dev/md/root level=raid1 num-devices=2 UUID=9ef8fef0:b3b04ded:94a9f448:2ac6f557
ARRAY /dev/md/swap level=raid1 num-devices=2 UUID=07cf2047:1d9f15c2:7e3cd6c5:ceb1e8fd

6. /etc/fstab/ before/after dnf upgrade (no difference):

# cat /etc/fstab
UUID=a7b1b7a1-f74c-46db-aee7-11662ed52529 /                       ext4    defaults        1 1
UUID=ddf4b3e4-9fd0-4179-a313-f7df55e75160 /boot                   ext4    defaults        1 2
UUID=3dac8ce5-7b99-4a91-aecf-39bff7199860 none                    swap    defaults        0 0

7. /proc/mdstat/ before/after dnf upgrade (differences in which array gets assigned which /dev/mdX, but that's expected):

# cat /proc/mdstat #<----------------------- BEFORE DNF UPGRADE
Personalities : [raid1] 
md125 : active raid1 sda5[0] sdb5[1]
      224858112 blocks super 1.2 [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

md126 : active raid1 sdb3[1] sda3[0]
      1046528 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 sda2[0] sdb2[1]
      8379392 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>

# cat /proc/mdstat #<----------------------- AFTER DNF UPGRADE
Personalities : [raid1] 
md125 : active raid1 sda3[0] sdb3[1]
      1046528 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 sdb2[1] sda2[0]
      8379392 blocks super 1.2 [2/2] [UU]
      
md127 : active raid1 sda5[0] sdb5[1]
      224858112 blocks super 1.2 [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

unused devices: <none>

8. List of UUIDs:

# blkid
/dev/sda1: PARTUUID="c7550c31-01"
/dev/sda2: UUID="07cf2047-1d9f-15c2-7e3c-d6c5ceb1e8fd" UUID_SUB="511cd23a-166a-4df9-7c32-6cb83c32cae9" LABEL="fedora:swap" TYPE="linux_raid_member" PARTUUID="c7550c31-02"
/dev/sda3: UUID="48134451-083e-8a98-a3b3-ff946c7ef07c" UUID_SUB="13989038-dba5-8a00-0af8-0acd2ac3ecbd" LABEL="fedora:boot" TYPE="linux_raid_member" PARTUUID="c7550c31-03"
/dev/sda5: UUID="9ef8fef0-b3b0-4ded-94a9-f4482ac6f557" UUID_SUB="7be0fb43-4ce8-7f22-8944-c68e96319568" LABEL="fedora:root" TYPE="linux_raid_member" PARTUUID="c7550c31-05"
/dev/sdb1: PARTUUID="397b4df3-01"
/dev/sdb2: UUID="07cf2047-1d9f-15c2-7e3c-d6c5ceb1e8fd" UUID_SUB="96ff5dc1-8a1d-f0ad-b49f-517508d04a9f" LABEL="fedora:swap" TYPE="linux_raid_member" PARTUUID="397b4df3-02"
/dev/sdb3: UUID="48134451-083e-8a98-a3b3-ff946c7ef07c" UUID_SUB="9622f2e0-c632-725d-c457-24556eeda5bd" LABEL="fedora:boot" TYPE="linux_raid_member" PARTUUID="397b4df3-03"
/dev/sdb5: UUID="9ef8fef0-b3b0-4ded-94a9-f4482ac6f557" UUID_SUB="eb7af984-8e27-cc8c-53bd-08e2f9fcc65d" LABEL="fedora:root" TYPE="linux_raid_member" PARTUUID="397b4df3-05"
/dev/md127: UUID="3dac8ce5-7b99-4a91-aecf-39bff7199860" TYPE="swap"
/dev/md126: UUID="ddf4b3e4-9fd0-4179-a313-f7df55e75160" BLOCK_SIZE="4096" TYPE="ext4"
/dev/md125: UUID="a7b1b7a1-f74c-46db-aee7-11662ed52529" BLOCK_SIZE="4096" TYPE="ext4"
/dev/zram0: LABEL="zram0" UUID="780c7be9-de22-4170-a586-4a72d9c22d92" TYPE="swap"

[zacharyfreeman70]

I see there's been no activity on this for over a year... is this the right location to be reporting these types of issues? I'm still running into this issue a year later.

[LaszloGombos]

CC @marcosfrm for ideas if any