lvm disk no longer bootable

[mranweil]

After updating the firmware from op920 I no longer am able to boot to a Ubuntu 18.04 installation that's on an LVM disk.

Petitboot (v1.10.2) 8335-GTW 13155BA
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
[Disk: sda2 / ad850f99-78ba-4f19-87ba-581802c44d34]
Red Hat Enterprise Linux Server (0-rescue-dd6ab4ad6f1b48b1956853e8a61a5df2)
Red Hat Enterprise Linux Server (4.14.0-49.el7a.ppc64le) 7.5 (Maipo)

*System information

From the system information:
Storage devices
sdc1:
UUID: 801E-9E95
mounted at: /var/petitboot/mnt/dev/sdc1

sda2:
UUID: ad850f99-78ba-4f19-87ba-581802c44d34
mounted at: /var/petitboot/mnt/dev/sda2

Primary platform versions:
IBM-witherspoon-OP9-v2.2-9.56
op-build-v2.2-14-g29f0ba6
buildroot-2018.11.3-12-g222837a
skiboot-v6.2-176-g261ca8e779e5
hostboot-0b3f4e1-p64efa43
occ-12c8088
linux-4.19.26-openpower1-p278e5c6
petitboot-v1.10.2
machine-xml-e3e9aef
hostboot-binaries-hw021419a.930
capp-ucode-p9-dd2-v4
sbe-1410677
And from the shell:

fdisk -l

Found valid GPT with protective MBR; using GPT

Disk /dev/sdb: 1875385008 sectors, 2307M
Logical sector size: 512
Disk identifier (GUID): 66eb5985-35f8-4b7b-ba46-92f1816b1655
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1875384974

Number Start (sector) End (sector) Size Name
1 2048 16383 7168K
2 16384 1875384319 894G
Disk /dev/sda: 894 GB, 960197124096 bytes, 1875385008 sectors
116737 cylinders, 255 heads, 63 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type
/dev/sda1 * 0,32,33 0,162,34 2048 10239 8192 4096K 41 PPC PReP Boot
/dev/sda2 0,162,35 131,45,42 10240 2107391 2097152 1024M 83 Linux
/dev/sda3 131,45,43 1023,254,63 2107392 1875384319 1873276928 893G 8e Linux LVM
Disk /dev/sdc: 3854 MB, 4041211904 bytes, 7892992 sectors
491 cylinders, 255 heads, 63 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type
/dev/sdc1 0,2,3 490,254,63 128 7889023 7888896 3852M c Win95 FAT32 (LBA)
Partition 1 has different physical/logical end:
phys=(490,254,63) logical=(491,17,38)

sdb was previously bootable, there's an LVM install there. This is similar to issue 58 - Manoj hit that with the same update as here, but our systems and symptoms are different, even though they both cause us boot issues.

[sammj]

I think you sent me one already but just for completeness lease run pb-sos in the shell and attach the result :)

#58 is different in that the boot option exists but an IPMI override is present for some reason. I suspect what you're going to see in the log is

[21:30:42] Running command:
 exe:  /usr/sbin/vgscan
 argv: '/usr/sbin/vgscan' '-qq'
  io_setup failed
  Failed to create bcache io engine.

Which leans towards either an issue with LVM or the kernel. I think I remember there being an issue around there recently, I'll see if I can recreate.

[sammj]

Indeed this may be the issue described here: https://bugzilla.redhat.com/show_bug.cgi?id=1650652
Our latest build uses the same LVM version, will test with an update.

[mranweil]

Or is that something you can make available and I can try, too? I didn't think this had been recreated....

Attaching the logs.
pb-sos.tar.gz

[sammj]

I've sent a patch to Buildroot to update to the latest lvm2 stable release, which we'll pick up in our Buildroot tree as well: https://patchwork.ozlabs.org/patch/1060738/

[sammj]

That's now in Buildroot master; we'll carry it in the op-build tree for a moment while we wait for a 2019.02 branch update.

[sammj]

Included in open-power/op-build#2732