Mount RHCOS LUKS

[cgwalters]

We need the internal libguestfs bits to work; at least
buildextend-installer uses them.

[cgwalters]

So I am still banging my head against this:

guestfsd: <= luks_open (0x101) request length 84 bytes
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: udevadm --debug settle -E /dev/sda4
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: cryptsetup -d /tmp/lukskwfgQY luksOpen /dev/sda4 crypt_root
guestfsd: error: 
guestfsd: => luks_open (0x101) took 0.04 secs
libguestfs: error: luks_open: 
libguestfs: closing guestfs handle 0x5599e70307e0 (state 2)

As best I can tell from reading the libguestfs source, that's saying that cryptsetup exited with nonzero, but emitted nothing to stderr. Or there's a bug in libguestfs.

[cgwalters]

OK, built libguestfs from source and hacked in prefixing cryptsetup with strace and got:

uname({sysname="Linux", nodename="(none)", ...}) = 0
stat("/dev/mapper/control", {st_mode=S_IFCHR|0600, st_rdev=makedev(0xa, 0xec), ...}) = 0
openat(AT_FDCWD, "/dev/mapper/control", O_RDWR) = 6
openat(AT_FDCWD, "/proc/devices", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(7, "Character devices:\n  1 mem\n  4 /"..., 1024) = 563
close(7)                                = 0
ioctl(6, DM_VERSION, {version=4.0.0, data_size=16384, flags=DM_EXISTS_FLAG} => {version=4.40.0, data_size=16384, flags=DM_EXISTS_FLAG}) = 0
ioctl(6, DM_LIST_VERSIONS, {version=4.1.0, data_size=16384, data_start=312, flags=DM_EXISTS_FLAG} => {version=4.40.0, data_size=561, data_start=312, flags=DM_EXISTS_FLAG, ...}) = 0
semctl(0, 0, SEM_INFO, 0x7ffcbb59cf10)  = 0
access("/run/udev/control", F_OK)       = 0
ioctl(6, DM_TABLE_STATUS, {version=4.0.0, data_size=16384, data_start=312, name="luks-00000000-0000-4000-a000-000000000002", flags=DM_EXISTS_FLAG|DM_NOFLUSH_FLAG} => {version=4.40.0, data_size=16384, data_start=312, name="luks-00000000-0000-4000-a000-000000000002", flags=DM_EXISTS_FLAG|DM_NOFLUSH_FLAG}) = -1 ENXIO (N

Something about devmapper not being fully setup or something?

[cgwalters]

So yeah, this took me a whole lot of time to debug because it was going OOM, but that was not at all obvious. See https://gitlab.com/cryptsetup/cryptsetup/issues/372

[cgwalters]

Lifting WIP on this, tested with both FCOS and RHCOS.

[cgwalters]

Hm, actually this reveals that the memory we pass to qemu for create_disk.sh "leaks" into the default --pbkdf-memory - cryptsetup is (rationally) assuming that one is opening the LUKS device on the same machine as one created it.

I'm going to play with an explicit --pbkdf-memory 524288 or so too, although we almost certainly will still need to bump the libguestfs memory anyways.

[darkmuggle]

LGTM, and very similar to what I fleshed out.

[darkmuggle]

Merging, this is _very similar to #830. I'm merging and putting another PR for the other improvements.