Fedora36 Next Auto-Upgrade Fails & Yields Grub Prompt

[fifofonix]

Describe the bug

Next node created on vSphere from OVA based on 35.20220313.1.0 with rpm-ostree-layered open-vm-tools failed to upgrade cleanly to 36.20220325.1.0. Identical sister node upgraded without issue.

Messages on the console seemed to indicate that kernel was found and a boot did commence but the machine did not come online from a network perspective. A hard reset of the VM via vSphere yielded the grub prompt.

Including systemd logs immediately prior to reboot into fedora36 in case these point to anything.

Reproduction steps

Unable to reproduce at this point.

System details

• vSphere
• Fedora CoreOS version 35.20220313.1.0 transitioning to 36.20220325.1.0.

Systemd Logs Prior To Reboot

Top is latest timestamp.

2022-03-30 14:54:53 | Created new deployment /ostree/deploy/fedora-coreos/deploy/bdb5195673a010315644f74fc1c195dd0356b5a371032e36155949eb53f180f7.0
-- | --
  |   | 2022-03-30 14:54:52 | audit: type=1130 audit(1648666492.468:4071): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ostree-finalize-staged comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
  |   | 2022-03-30 14:54:52 | Finished OSTree Finalize Staged Deployment.
  |   | 2022-03-30 14:54:52 | SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ostree-finalize-staged comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
  |   | 2022-03-30 14:54:52 | note: Deploying commit bdb5195673a010315644f74fc1c195dd0356b5a371032e36155949eb53f180f7 which contains content in /var/lib that will be ignored.
  |   | 2022-03-30 14:54:50 | Wrote commit: bdb5195673a010315644f74fc1c195dd0356b5a371032e36155949eb53f180f7; New objects: meta:67 content:2 totaling 16.3 MB)
  |   | 2022-03-30 14:54:42 | Regenerating rpmdb for target
  |   | 2022-03-30 14:54:42 | sanitycheck(/usr/bin/true) successful
  |   | 2022-03-30 14:54:42 | Executed %transfiletriggerin(systemd-udev) for usr/lib/udev/rules.d in 53 ms; 73 matched files
  |   | 2022-03-30 14:54:41 | Executed %transfiletriggerin(systemd-udev) for usr/lib/udev/hwdb.d in 39 ms; 28 matched files
  |   | 2022-03-30 14:54:41 | Executed %transfiletriggerin(shared-mime-info) for usr/share/mime in 63 ms; 824 matched files
  |   | 2022-03-30 14:54:41 | No files matched %transfiletriggerin(usr/share/glib-2.0/schemas) for glib2
  |   | 2022-03-30 14:54:41 | No files matched %transfiletriggerin(usr/lib64/gio/modules) for glib2
  |   | 2022-03-30 14:54:41 | Executed %transfiletriggerin(glibc-common) for lib, lib64, usr/lib, usr/lib64 in 217 ms; 12038 matched files
  |   | 2022-03-30 14:54:41 | No files matched %transfiletriggerin(lib64) for glibc-common
  |   | 2022-03-30 14:54:41 | No files matched %transfiletriggerin(lib) for glibc-common
  |   | 2022-03-30 14:54:41 | Executed %post for open-vm-tools in 266 ms
  |   | 2022-03-30 14:54:41 | Created symlink /etc/systemd/system/vmtoolsd.service.requires/vgauthd.service → /usr/lib/systemd/system/vgauthd.service.
  |   | 2022-03-30 14:54:41 | Created symlink /etc/systemd/system/multi-user.target.wants/vmtoolsd.service → /usr/lib/systemd/system/vmtoolsd.service.
  |   | 2022-03-30 14:54:41 | Imported 10 pkgs
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/h/hwdata-0.357-1.fc36.noarch.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/l/libdrm-2.4.110-1.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/l/libmspack-0.10.1-0.7.alpha.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/l/libpciaccess-0.16-6.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/l/libxslt-1.1.35-1.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/o/open-vm-tools-11.3.5-1.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/p/pciutils-3.7.0-5.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/p/pciutils-libs-3.7.0-5.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/x/xmlsec1-1.2.33-2.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:40 | Downloading: http://mirrors.mit.edu/fedora/linux/development/36/Everything/x86_64/os/Packages/x/xmlsec1-openssl-1.2.33-2.fc36.x86_64.rpm
  |   | 2022-03-30 14:54:39 | Downloading: https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64
  |   | 2022-03-30 14:54:39 | Preparing pkg txn; enabled repos: ['fedora-cisco-openh264', 'fedora-modular', 'updates-modular', 'updates', 'fedora', 'updates-archive'] solvables: 68770
  |   | 2022-03-30 14:54:18 | Downloading: https://fedoraproject-updates-archive.fedoraproject.org/fedora/36/x86_64/repodata/2e7db4492173b6c437fd1299dc335e63d09f24cbdadeac5175a61b787c2f7a44-filelists.xml.zck
  |   | 2022-03-30 14:54:18 | Downloading: https://fedoraproject-updates-archive.fedoraproject.org/fedora/36/x86_64/repodata/4a5f5a36381e29d9a70665b60eab96d702e5fb5c1a61904cea685dd5d83df2d0-primary.xml.zck
  |   | 2022-03-30 14:54:18 | Downloading: https://fedoraproject-updates-archive.fedoraproject.org/fedora/36/x86_64/repodata/2e7db4492173b6c437fd1299dc335e63d09f24cbdadeac5175a61b787c2f7a44-filelists.xml.zck
  |   | 2022-03-30 14:54:18 | Downloading: https://fedoraproject-updates-archive.fedoraproject.org/fedora/36/x86_64/repodata/4a5f5a36381e29d9a70665b60eab96d702e5fb5c1a61904cea685dd5d83df2d0-primary.xml.zck
  |   | 2022-03-30 14:54:18 | Downloading: https://fedoraproject-updates-archive.fedoraproject.org/fedora/36/x86_64/repodata/repomd.xml
  |   | 2022-03-30 14:54:18 | Downloading: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-36-x86_64
  |   | 2022-03-30 14:53:56 | Downloading: http://download-ib01.fedoraproject.org/pub/fedora/linux/development/36/Everything/x86_64/os/repodata/0d9f276e75f4eb5ca1ef161d079e420b36e8f208673a45b2c571ad99949ac81f-filelists.xml.zck
  |   | 2022-03-30 14:53:56 | Downloading: http://download-ib01.fedoraproject.org/pub/fedora/linux/development/36/Everything/x86_64/os/repodata/6f826637e48be881dfa9d9d1177ace19ff7c007dfd8a262142fac8cbb0c8a60c-primary.xml.zck
  |   | 2022-03-30 14:53:55 | Downloading: http://download-ib01.fedoraproject.org/pub/fedora/linux/development/36/Everything/x86_64/os/repodata/51291371fd4fe63da59d48f442b621c118c16a5c70019760b3277b920ed63296-comps-Everything.x86_64.xml.zck
  |   | 2022-03-30 14:53:55 | Downloading: http://download-ib01.fedoraproject.org/pub/fedora/linux/development/36/Everything/x86_64/os/repodata/51291371fd4fe63da59d48f442b621c118c16a5c70019760b3277b920ed63296-comps-Everything.x86_64.xml.zck
  |   | 2022-03-30 14:53:55 | Downloading: http://download-ib01.fedoraproject.org/pub/fedora/linux/development/36/Everything/x86_64/os/repodata/0d9f276e75f4eb5ca1ef161d079e420b36e8f208673a45b2c571ad99949ac81f-filelists.xml.zck
  |   | 2022-03-30 14:53:55 | Downloading: http://download-ib01.fedoraproject.org/pub/fedora/linux/development/36/Everything/x86_64/os/repodata/6f826637e48be881dfa9d9d1177ace19ff7c007dfd8a262142fac8cbb0c8a60c-primary.xml.zck
  |   | 2022-03-30 14:53:55 | Downloading: http://download-ib01.fedoraproject.org/pub/fedora/linux/development/36/Everything/x86_64/os/repodata/repomd.xml
  |   | 2022-03-30 14:53:55 | Downloading: https://mirrors.fedoraproject.org/metalink?repo=fedora-36&arch=x86_64
  |   | 2022-03-30 14:53:55 | Downloading: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-36-x86_64
  |   | 2022-03-30 14:53:55 | Downloading: http://mirror.arizona.edu/fedora/linux/updates/36/Everything/x86_64/repodata/95a4415d859d7120efb6b3cf964c07bebbff9a5275ca673e6e74a97bcbfb2a5f-filelists.xml.gz
  |   | 2022-03-30 14:53:55 | Downloading: http://mirror.arizona.edu/fedora/linux/updates/36/Everything/x86_64/repodata/1cb61ea996355add02b1426ed4c1780ea75ce0c04c5d1107c025c3fbd7d8bcae-primary.xml.gz
  |   | 2022-03-30 14:53:55 | Downloading: http://mirror.arizona.edu/fedora/linux/updates/36/Everything/x86_64/repodata/repomd.xml
  |   | 2022-03-30 14:53:54 | Downloading: https://mirrors.fedoraproject.org/metalink?repo=updates-released-f36&arch=x86_64
  |   | 2022-03-30 14:53:54 | Downloading: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-36-x86_64
  |   | 2022-03-30 14:53:54 | Downloading: https://mirror.umd.edu/fedora/linux/updates/36/Modular/x86_64/repodata/95a4415d859d7120efb6b3cf964c07bebbff9a5275ca673e6e74a97bcbfb2a5f-filelists.xml.gz
  |   | 2022-03-30 14:53:54 | Downloading: https://mirror.umd.edu/fedora/linux/updates/36/Modular/x86_64/repodata/1cb61ea996355add02b1426ed4c1780ea75ce0c04c5d1107c025c3fbd7d8bcae-primary.xml.gz
  |   | 2022-03-30 14:53:54 | Downloading: https://mirror.umd.edu/fedora/linux/updates/36/Modular/x86_64/repodata/repomd.xml
  |   | 2022-03-30 14:53:54 | Downloading: https://mirrors.fedoraproject.org/metalink?repo=updates-released-modular-f36&arch=x86_64
  |   | 2022-03-30 14:53:54 | Downloading: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-36-x86_64
  |   | 2022-03-30 14:53:51 | Downloading: https://mirrors.ocf.berkeley.edu/fedora/fedora/linux/development/36/Modular/x86_64/os/repodata/16407f791df4313ceef54ee4b3ec8a257fc81ace8ac70cf45eaa75166c78b37a-modules.yaml.gz
  |   | 2022-03-30 14:53:51 | Downloading: https://mirrors.ocf.berkeley.edu/fedora/fedora/linux/development/36/Modular/x86_64/os/repodata/e937baca7bf0eed9dad9f7b49adb5ef455a53e7f1b0b07ad0f3cea3c8590e897-filelists.xml.zck
  |   | 2022-03-30 14:53:51 | Downloading: https://mirrors.ocf.berkeley.edu/fedora/fedora/linux/development/36/Modular/x86_64/os/repodata/9e78a460381714530b0a14623f8f81f0edfdb0048d36d0dc98bd43ce58373817-primary.xml.zck
  |   | 2022-03-30 14:53:51 | Downloading: https://mirrors.ocf.berkeley.edu/fedora/fedora/linux/development/36/Modular/x86_64/os/repodata/bd0caa6f35c4034faf0bc433ca4dfba9bbf9004074da51fe4a0aeb7dc9f4caa4-comps-Modular.x86_64.xml.zck
  |   | 2022-03-30 14:53:50 | Downloading: https://mirrors.ocf.berkeley.edu/fedora/fedora/linux/development/36/Modular/x86_64/os/repodata/bd0caa6f35c4034faf0bc433ca4dfba9bbf9004074da51fe4a0aeb7dc9f4caa4-comps-Modular.x86_64.xml.zck
  |   | 2022-03-30 14:53:50 | Downloading: https://mirrors.ocf.berkeley.edu/fedora/fedora/linux/development/36/Modular/x86_64/os/repodata/e937baca7bf0eed9dad9f7b49adb5ef455a53e7f1b0b07ad0f3cea3c8590e897-filelists.xml.zck
  |   | 2022-03-30 14:53:50 | Downloading: https://mirrors.ocf.berkeley.edu/fedora/fedora/linux/development/36/Modular/x86_64/os/repodata/9e78a460381714530b0a14623f8f81f0edfdb0048d36d0dc98bd43ce58373817-primary.xml.zck
  |   | 2022-03-30 14:53:50 | Downloading: https://mirrors.ocf.berkeley.edu/fedora/fedora/linux/development/36/Modular/x86_64/os/repodata/repomd.xml
  |   | 2022-03-30 14:53:50 | Downloading: https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-36&arch=x86_64
  |   | 2022-03-30 14:53:50 | Downloading: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-36-x86_64
  |   | 2022-03-30 14:53:50 | Downloading: https://codecs.fedoraproject.org/openh264/36/x86_64/os/repodata/2b13cd3f9d81647fd31aa16de1b16b582efd9566f8c4334e4561a030f3777c37-comps-Temporary.x86_64.xml
  |   | 2022-03-30 14:53:50 | Downloading: https://codecs.fedoraproject.org/openh264/36/x86_64/os/repodata/b3eeffc8d4c4c8381c03954b3ab95bdbe89e6628f59ec4d2d6cb4c6f81d999ab-filelists.xml.gz
  |   | 2022-03-30 14:53:50 | Downloading: https://codecs.fedoraproject.org/openh264/36/x86_64/os/repodata/f317c383b2bdf0260eddd22565f7ee0661c0d4dd229a6b0858c8ad2c5767be5d-primary.xml.gz
  |   | 2022-03-30 14:53:49 | Downloading: https://codecs.fedoraproject.org/openh264/36/x86_64/os/repodata/repomd.xml
  |   | 2022-03-30 14:53:49 | Downloading: https://mirrors.fedoraproject.org/metalink?repo=fedora-cisco-openh264-36&arch=x86_64
  |   | 2022-03-30 14:53:49 | Downloading: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-36-x86_64
  |   | 2022-03-30 14:53:49 | Librepo version: 1.14.2 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.79.1 OpenSSL/1.1.1l-fips zlib/1.2.11 brotli/1.0.9 libidn2/2.3.2 libpsl/0.21.1 (+libidn2/2.3.2) libssh/0.9.6/openssl/zlib nghttp2/1.45.1 OpenLDAP/2.4.59)
  |   | 2022-03-30 14:53:48 | libostree pull from 'fedora' for fedora/x86_64/coreos/next complete security: GPG: commit  security: SIGN: disabled http: TLS non-delta: meta: 1932 content: 7761 transfer: secs: 109 size: 503.9 MB
  |   | 2022-03-30 14:53:40 | Receiving objects; 96% (9373/9693) 4.5 MB/s 451.7 MB
  |   | 2022-03-30 14:53:29 | run-docker-runtime\x2drunc-moby-7e4222bb4061afa671c67ddf7fe07eeef4b3f62ece42cc87606bc0e5f2837ac7-runc.uIg1tC.mount: Deactivated successfully.
  |   | 2022-03-30 14:53:23 | Receiving objects; 84% (8231/9693) 4.2 MB/s 351.0 MB
  |   | 2022-03-30 14:53:07 | W0330 18:53:07.904314       1 machine_libipmctl.go:64] There are no NVM devices!
  |   | 2022-03-30 14:53:07 | Receiving objects; 67% (6582/9693) 4.0 MB/s 269.7 MB
  |   | 2022-03-30 14:53:05 | SELinux:  Context system_u:object_r:NetworkManager_priv_helper_exec_t:s0 is not valid (left unmapped).
  |   | 2022-03-30 14:53:05 | SELinux:  Context system_u:object_r:NetworkManager_dispatcher_exec_t:s0 is not valid (left unmapped).
  |   | 2022-03-30 14:52:59 | SELinux:  Context system_u:object_r:NetworkManager_dispatcher_script_t:s0 is not valid (left unmapped).
  |   | 2022-03-30 14:52:50 | Receiving objects; 49% (4813/9693) 4.0 MB/s 198.1 MB
  |   | 2022-03-30 14:52:47 | SELinux:  Context system_u:object_r:NetworkManager_dispatcher_cloud_script_t:s0 is not valid (left unmapped).
  |   | 2022-03-30 14:52:33 | Receiving objects; 33% (3263/9693) 3.7 MB/s 123.0 MB
  |   | 2022-03-30 14:52:29 | run-docker-runtime\x2drunc-moby-7e4222bb4061afa671c67ddf7fe07eeef4b3f62ece42cc87606bc0e5f2837ac7-runc.mHK2bu.mount: Deactivated successfully.
  |   | 2022-03-30 14:52:21 | SELinux:  Context system_u:object_r:NetworkManager_dispatcher_chronyc_script_t:s0 is not valid (left unmapped).
  |   | 2022-03-30 14:52:17 | Receiving objects; 25% (2515/9693) 2.5 MB/s 41.8 MB
  |   | 2022-03-30 14:52:09 | SELinux:  Context system_u:object_r:NetworkManager_dispatcher_iscsid_script_t:s0 is not valid (left unmapped).
  |   | 2022-03-30 14:52:01 | Receiving metadata objects: 0/(estimating) -/s 0 bytes
  |   | 2022-03-30 14:51:59 | Process [pid: 1160377 uid: 980 unit: zincati.service] connected to transaction progress
  |   | 2022-03-30 14:51:59 | Initiated txn Deploy for client(dbus:1.571 unit:zincati.service uid:980): /org/projectatomic/rpmostree1/fedora_coreos
  |   | 2022-03-30 14:51:59 | Locked sysroot
  |   | 2022-03-30 14:51:59 | ext4 filesystem being remounted at /boot supports timestamps until 2038 (0x7fffffff)
  |   | 2022-03-30 14:51:59 | EXT4-fs (sda3): re-mounted. Opts: . Quota mode: none.
  |   | 2022-03-30 14:51:59 | xfs filesystem being remounted at /sysroot supports timestamps until 2038 (0x7fffffff)
  |   | 2022-03-30 14:51:59 | audit: type=1130 audit(1648666319.815:4070): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpm-ostreed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
  |   | 2022-03-30 14:51:59 | SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpm-ostreed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
  |   | 2022-03-30 14:51:59 | Started rpm-ostree System Management Daemon.
  |   | 2022-03-30 14:51:59 | In idle state; will auto-exit in 64 seconds
  |   | 2022-03-30 14:51:59 | Reading config file '/etc/rpm-ostreed.conf'
  |   | 2022-03-30 14:51:59 | Starting rpm-ostree System Management Daemon...
  |   | 2022-03-30 14:51:59 | [INFO  zincati::update_agent::actor] target release '36.20220325.1.0' selected, proceeding to stage it
  |   | 2022-03-30 14:50:01 | time="2022-03-30T18:50:01.736282653Z" level=info msg="NetworkDB stats d-pdm-is-2(5226b3d9e2eb) - netID:awgmc7h6i8rab36tblzd1bkkj leaving:false netPeers:2 entries:4 Queue qLen:0 netMsg/s:0"

[fifofonix]

Select details from the grub console gathered with @dustymabe:

Initial failed boot message:

�[H�[J�[1;1Herror: ../../grub-core/commands/boot.c:146:you need to load the kernel first.

�[H�[J�[1;1H�[?25h      Minimal BASH-like line editing is supported. For the first word,    

      TAB lists possible command completions. Anywhere else TAB lists     

      possible device or file completions.                                

The results of ls (hd0,gpt3)/loader/entries/ shows that there is no such file. It turns out in this instance that (hd0,gpt3)/loader is a symlink to itself and that there is instead a (hd0,gpt3)/loader.tmp file with the entries subdirectory implying some kind of interim failure.

grub> �[6;7Hl�[6;8H�[6;8Hl�[6;9H�[6;8H�[6;8H �[6;8H�[6;8H�[6;8Hs�[6;9H�[6;9H �[6;10H�[6;10H(�[6;11H�[6;11Hh�[6;12H�[6;12Hd�[6;13H�[6;13H0�[6;14H�[6;14H,�[6;15H�[6;15Hg�[6;16H�[6;16Hp�[6;17H�[6;17Ht�[6;18H�[6;18H3�[6;19H�[6;19H)�[6;20H�[6;20H/�[6;21H�[6;21Hl�[6;22H�[6;22Ho�[6;23H�[6;23Ha�[6;24H�[6;24Hd�[6;25H�[6;25He�[6;26H�[6;26Hr�[6;27H�[6;27H/�[6;28H�[6;28He�[6;29H�[6;29Hn�[6;30H�[6;30Ht�[6;31H�[6;31Hr�[6;32H�[6;32Hi�[6;33H�[6;33He�[6;34H�[6;34Hs�[6;35H�[6;35H/�[6;36H


error: ../../grub-core/fs/fshelp.c:257:file `/loader/entries/' not found.

grub> �[9;7H�[9;7H�[9;7H�[9;7Hls (hd0,gpt3)/loader/entries/�[9;36H�[9;35H�[9;35H �[9;35H�[9;35H�[9;34H�[9;34H �[9;34H�[9;34H�[9;33H�[9;33H �[9;33H�[9;33H�[9;32H�[9;32H �[9;32H�[9;32H�[9;31H�[9;31H �[9;31H�[9;31H�[9;30H�[9;30H �[9;30H�[9;30H�[9;29H�[9;29H �[9;29H�[9;29H�[9;28H�[9;28H �[9;28H�[9;28H�[9;27H�[9;27H �[9;27H�[9;27H�[9;27H.�[9;28H�[9;28H1�[9;29H�[9;29H/�[9;30H�[9;30He�[9;31H�[9;31Hn�[9;32H�[9;32Ht�[9;33H�[9;33Hr�[9;34H�[9;34Hi�[9;35H�[9;35He�[9;36H�[9;36Hs�[9;37H�[9;37H/�[9;38H

./ ../ ostree-2-fedora-coreos.conf ostree-1-fedora-coreos.conf 

grub> �[11;7H�[11;7H�[11;7H�[11;7Hls (hd0,gpt3)/loader.1/entries/�[11;38H�[11;37H�[11;37H �[11;37H�[11;37H�[11;36H�[11;36H �[11;36H�[11;36H�[11;35H�[11;35H �[11;35H�[11;35H�[11;34H�[11;34H �[11;34H�[11;34H�[11;33H�[11;33H �[11;33H�[11;33H�[11;32H�[11;32H �[11;32H�[11;32H�[11;31H�[11;31H �[11;31H�[11;31H�[11;30H�[11;30H �[11;30H�[11;30H�[11;29H�[11;29H �[11;29H�[11;29H�[11;28H�[11;28H �[11;28H�[11;28H�[11;27H�[11;27H �[11;27H�[11;27H�[11;26H�[11;26H �[11;26H�[11;26H�[11;25H�[11;25H �[11;25H�[11;25H�[11;24H�[11;24H �[11;24H�[11;24H�[11;23H�[11;23H �[11;23H�[11;23H�[11;22H�[11;22H �[11;22H�[11;22H�[11;21H�[11;21H �[11;21H�[11;21H

./ ../ lost+found/ efi/ loader.1/ ostree/ boot loader grub2/ bootupd-state.json

loader.0/ .root_uuid loader.tmp 

grub> �[14;7H+�[14;8H�[14;7H�[14;7H �[14;7H�[14;7H�[14;7H+�[14;8H�[14;7H�[14;7H �[14;7H�[14;7H�[14;7Hl�[14;8H�[14;8Hs�[14;9H�[14;9H �[14;10H�[14;10H(�[14;11H�[14;11Hh�[14;12H�[14;12Hd�[14;13H�[14;13H-�[14;14H�[14;13H�[14;13H �[14;13H�[14;13H�[14;7H      �[14;7H�[14;7H�[14;7Hls (hd0,gpt3)/�[14;21H�[14;21Hl�[14;22H�[14;22Ho�[14;23H�[14;23Ha�[14;24H�[14;24Hd�[14;25H�[14;25He�[14;26H�[14;26Hr�[14;27H

./ ../ lost+found/ efi/ loader.1/ ostree/ boot loader grub2/ bootupd-state.json

loader.0/ .root_uuid loader.tmp 



grub> �[24;7H�[24;7H�[24;7H�[24;7Hhelp ls�[24;14H�[24;7H       �[24;7H�[24;7H�[24;7Hhelp ks�[24;14H�[24;7H       �[24;7H�[24;7H�[24;7Hls -l (hd0,gpt3)/loader�[24;30H�[24;29H�[24;28H�[24;27H�[24;26H�[24;25H�[24;24H�[24;23H�[24;22H�[24;21H�[24;20H�[24;19H�[24;18H�[24;17H�[24;16H�[24;15H�[24;14H�[24;13H�[24;12H�[24;11H�[24;12H�[24;11H�[24;29H �[24;11H (hd0,gpt3)/loader�[24;11H�[24;10H�[24;28H �[24;10H (hd0,gpt3)/loader�[24;10H�[24;9H�[24;27H �[24;9H (hd0,gpt3)/loader�[24;9H

./ ../ lost+found/ efi/ loader.1/ ostree/ boot loader grub2/ bootupd-state.json

loader.0/ .root_uuid loader.tmp 

grub> �[24;7H�[24;7H�[24;7H�[24;7Hls (hd0,gpt3)/loader�[24;27H�[24;27H/�[24;28H

./ ../ lost+found/ efi/ loader.1/ ostree/ boot loader grub2/ bootupd-state.json

loader.0/ .root_uuid loader.tmp 

grub> �[24;7H�[24;7H�[24;7H�[24;7Hls (hd0,gpt3)/loader/�[24;28H�[24;28Hl�[24;29H�[24;29Ho�[24;30H�[24;30Ha�[24;31H�[24;31Hd�[24;32H�[24;32He�[24;33H�[24;33Hr�[24;34H

./ ../ lost+found/ efi/ loader.1/ ostree/ boot loader grub2/ bootupd-state.json

loader.0/ .root_uuid loader.tmp 

grub> 

grub> �[24;7HK�[24;8H�[24;8HK�[24;9H�[24;9HK�[24;10H�[24;9H�[24;9H �[24;9H�[24;9H�[24;8H�[24;8H �[24;8H�[24;8H�[24;7H�[24;7H �[24;7H�[24;7H�[24;7HL�[24;8H�[24;8HS�[24;9H�[24;8H�[24;8H �[24;8H�[24;8H�[24;7H�[24;7H �[24;7H�[24;7H�[24;7Hl�[24;8H�[24;8Hs�[24;9H�[24;9H �[24;10H�[24;10H(�[24;11H�[24;11HH�[24;12H�[24;12HD�[24;13H�[24;13H0�[24;14H�[24;13H�[24;13H �[24;13H�[24;13H�[24;12H�[24;12H �[24;12H�[24;12H�[24;11H�[24;11H �[24;11H�[24;11H�[24;11Hh�[24;12H�[24;12Hd�[24;13H�[24;13H0�[24;14H�[24;14H,�[24;15H�[24;15Hg�[24;16H�[24;16Hp�[24;17H�[24;17Ht�[24;18H�[24;18H3�[24;19H�[24;19H)�[24;20H�[24;20H/�[24;21H�[24;21Hl�[24;22H�[24;22Ho�[24;23H�[24;23Ha�[24;24H�[24;24Hd�[24;25H�[24;25He�[24;26H�[24;26Hr�[24;27H�[24;27H/�[24;28H�[24;28Hl�[24;29H�[24;29Ho�[24;30H�[24;30Ha�[24;31H�[24;31Hd�[24;32H�[24;32He�[24;33H�[24;33Hr�[24;34H�[24;34H.�[24;35H�[24;35Ht�[24;36H�[24;36Hm�[24;37H�[24;37Hp�[24;38H�[24;37H�[24;36H�[24;35H�[24;34H�[24;33H�[24;32H�[24;31H�[24;30H�[24;29H�[24;28H�[24;27H�[24;26H�[24;25H�[24;24H�[24;23H�[24;24H�[24;25H�[24;26H�[24;27H�[24;26H�[24;37H �[24;26H/loader.tmp�[24;26H�[24;25H�[24;36H �[24;25H/loader.tmp�[24;25H�[24;24H�[24;35H �[24;24H/loader.tmp�[24;24H�[24;23H�[24;34H �[24;23H/loader.tmp�[24;23H�[24;22H�[24;33H �[24;22H/loader.tmp�[24;22H�[24;21H�[24;32H �[24;21H/loader.tmp�[24;21H�[24;20H�[24;31H �[24;20H/loader.tmp�[24;20H

./ ../ entries/ 

grub> 

[dustymabe]

Yep. Basically what we saw was:

• loader was a symlink to the current directory
  • loader -> .
• a loader.tmp directory existed that pointed to either loader.1 or loader.0 (can't remember)
  • loader.tmp -> loader.1
• loader.1 and loader.0 directories both had an entries folder (as they should)

Since the loader symlink was wrong blscfg (at the bottom of grub.cfg) didn't work and when we used configfile (hd0,gpt3)/grub2/grub.cfg would just complain saying you need to load the kernel first and dropping us back to the grub prompt.

So the question is: "How did the symlink manage to get messed up?"

[jlebon]

I'm not sure, but this looks like the boot filesystem was corrupted during the hard reset. So possibly an ext4 issue? At no point is loader a symlink to . in the code, so this doesn't represent any possible intermediate state.

[dustymabe]

Is there any code that creates a loader.tmp symlink? That could be another clue to look for.