BTRFS: grubby fatal error: unable to find a suitable template

[cmurf]

grubby-8.40-3.fc24.x86_64

Summary:
When a Btrfs subvolume is mounted at /boot, grubby fails to update grub.cfg with this error message. grub2-mkconfig has no issue with this configuration and produces working grub.cfg.

Detail:

[chris@localhost ~]$ cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Thu Jan 19 15:38:36 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=6b187fa6-f62a-47f8-b09c-0bcbc7a7036e /                       btrfs   subvol=root     0 0
UUID=6b187fa6-f62a-47f8-b09c-0bcbc7a7036e /home                   btrfs   subvol=home     0 0
UUID=ee093fa4-8d28-4169-83c8-ff5c998f599e swap                    swap    defaults        0 0

[chris@localhost ~]$ sudo blkid
/dev/vda1: UUID="9c6af6bb-a87c-4881-8406-e2e5abd5efa5" TYPE="ext4" PARTUUID="361cd7e6-01"
/dev/vda2: UUID="ee093fa4-8d28-4169-83c8-ff5c998f599e" TYPE="swap" PARTUUID="361cd7e6-02"
/dev/vda3: LABEL="fedora" UUID="6b187fa6-f62a-47f8-b09c-0bcbc7a7036e" UUID_SUB="355f5c4d-952a-4d03-a793-74accbe14540" TYPE="btrfs" PARTUUID="361cd7e6-03"

[chris@localhost ~]$ stat /
  File: /
  Size: 154       	Blocks: 0          IO Block: 4096   directory
Device: 27h/39d	Inode: 256         Links: 1
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:root_t:s0
Access: 2017-01-19 13:38:39.723713159 -0700
Modify: 2017-01-10 04:12:29.516524841 -0700
Change: 2017-01-19 13:30:47.771278980 -0700
 Birth: -

[chris@localhost ~]$ stat /boot
  File: /boot
  Size: 790       	Blocks: 0          IO Block: 4096   directory
Device: 27h/39d	Inode: 257         Links: 1
Access: (0555/dr-xr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:boot_t:s0
Access: 2017-01-19 13:55:11.488599883 -0700
Modify: 2017-01-19 13:55:10.570588940 -0700
Change: 2017-01-19 13:55:10.570588940 -0700
 Birth: -

[chris@localhost ~]$ cat /proc/mounts   ##summary
/dev/vda3 / btrfs rw,seclabel,relatime,space_cache,subvolid=257,subvol=/root 0 0
/dev/vda3 /home btrfs rw,seclabel,relatime,space_cache,subvolid=258,subvol=/home 0 0

Evaluation:
This example uses /boot as a directory in the 'root' subvolume which is in turn mounted at /.

Basically this bug is not so much a Btrfs subvolume problem per se, but a combination of how Fedora's installer creates subvolumes at the top level of the Btrfs volume, and how they end up being bind mounted behind the scenes with the Btrfs subvol= mount option found in fstab. GRUB understands this by always following the top level of the volume whether it's mounted or not. Grubby doesn't seem to know about the top level of Btrfs volumes, it merely follows the mounted path structure. Therefore it doesn't recognize the path /root/boot/ instead it only sees the path as /boot/.

An old downstream bug report uses a 'boot' subvolume mounted at /boot which runs into the identical error.
https://bugzilla.redhat.com/show_bug.cgi?id=864198

grub.cfg.txt
grubby-debug.log.txt

[rbauduin]

A similar problem occurs when using raid, and having the root partition (with /boot) on md0. grubby can't determine the uuid of the root partition. It chokes at this line: https://github.com/rhinstaller/grubby/blob/master/grubby.c#L2191
On a server where grubby works fine, strace reports this:

open("/sys/dev/block/253:1", O_RDONLY|O_CLOEXEC) = 4
newfstatat(4, "partition", {st_mode=S_IFREG|0444, st_size=4096, ...}, 0) = 0

On a server with the problem, strace has this:

open("/sys/dev/block/9:1", O_RDONLY|O_CLOEXEC) = 4
newfstatat(4, "partition", 0x7c4f936b5640, 0) = -1 ENOENT (No such file or directory)
openat(4, "dm/uuid", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
close(4)                                = 0

[jtriley]

Just a note that I'm also getting this when running yum upgrade inside a CentOS 6 qcow2 image (http://cloud.centos.org/centos/6/images/CentOS-6-x86_64-GenericCloud.qcow2) from a guestfish session using libguestfs. Running grubby after booting the same qcow2 image outside of libguestfs doesn't have the issue. I'm suspecting it's something to do with blkid version, kernel version, or some combo of the two and the fact that there's a ~4gb qcow2 appliance device visible in the OS only when I'm inside the guestfish session (this /dev/sdb device isn't present in a "real" boot):

><fs> command 'fdisk -l'

Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000566fa

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        3917    31455104   83  Linux

Disk /dev/sdb: 4294 MB, 4294967296 bytes
133 heads, 62 sectors/track, 1017 cylinders
Units = cylinders of 8246 * 512 = 4221952 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

It's worth noting that /dev/sdb isn't mounted anywhere.

Seems another libguestfs user had some similar issue back in 2010 https://www.redhat.com/archives/libguestfs/2010-April/msg00079.html. They worked around this by just generating the grub.conf update themselves in this case using augeas. Excerpt:

When installing kernel-2.6.18-128.el5 in a guest with
kernel-xen-2.6.18-164.el5xen already installed, and 'root=LABEL=/' on the kernel
command line, grubby fails to install a grub entry for the new kernel. It only
fails when run under libguestfs. Installing in the original Xen guest works
fine. The underlying cause appears to be related to blkid, although I haven't
managed to pin it down exactly.

This change works around the issue. If virt-v2v encounters a problem with grub,
it will install a new kernel entry itself using augeas.

In case it's useful to others my workaround is to run the following hack after yum upgrade in my bootstrap scripts:

#!/bin/bash
MAJ_DIST_RELEASE=$(rpm -q --queryformat '%{VERSION}' centos-release)

# fix grub.conf to boot latest kernel (grubby fails on upgrade)
if [ $MAJ_DIST_RELEASE -eq '6' ]; then
  kernel_rpm=$(rpm -q kernel | sort -V | tail -1)
  kernel=$(rpm -ql $kernel_rpm | grep -i /boot/vmlinuz)
  initrd=$(rpm -ql $kernel_rpm | grep -i /boot/initramfs)
  kernel_version=$(rpm -q $kernel_rpm --queryformat "%{VERSION}-%{RELEASE}.%{ARCH}")
  grubby --title="CentOS 6 ($kernel_version)" --add-kernel=$kernel --initrd=$initrd --make-default --copy-default --bad-image-okay
fi

This is essentially replicating the grubby post installation step of the kernel package but adding in --bad-image-okay which is obviously less than ideal but works.

FWIW I dont seem to experience the issue with the CentOS 7 qcow2 images...

[actatux]

I hit the same issue than @jtriley with a RHEL 6 cloud image.

Looking at grubby code (branch rhel-6.8), it looks like the grubby command is failing in findDiskForRoot(). This function search the drive in /etc/mtab.

I found out the / device is /dev/vda1 in /etc/mtab while it is /dev/sda1 in the guestfish session.

While only relevant to the previous comment, a basic s/vda1/sda1/ in /etc/mtab solves this issue for EL6 cloud images with guestfish. (or --edit "/etc/mtab:s/vda1/sda1/" in virt-sysprep/virt-customize)

[cmurf]

I think the error message is a generic one, so everyone with configurations other than the original bug report, which is about Btrfs subvolumes, needs to file separate bugs.

[herrold]

I don't know that this is BTRFS specific so much as 'no separate /boot' mount specific Of course a non-readable FS adds to the friction, but any failure to be able to read (no FS support, or simple non-presence) is enough

I have an open bug on this at Red Hat presently

https://bugzilla.redhat.com/show_bug.cgi?id=1498169

[cmurf]

It's definitely not 'no separate /boot' related because I've done /boot on LVM, /boot on mdadm raid of various levels, /boot as a directory part of rootfs, where rootfs is a range of simple to complex. As I understand it, grubby is looking at the kernel command line, and tests it against templates for layouts it's explicitly supporting and if there's something about the layout it isn't familiar with (no template) then it fails with this error message.

[neilbags]

hit this problem today. my current workaround is to move /usr/sbin/grubby /usr/sbin/grubby.real and make a wrapper that looks like looks like:

/usr/sbin/grubby.real --bad-image-okay "$@"

[thynson]

Please check #34, it does not work on my machine yet.