40ignition-ostree: add coreos-inject-rootmap.service #503

jlebon · 2020-06-29T21:15:50Z

This implements the rootmap functionality that figures out all the
dependencies required to find /sysroot, and injects them into the BLS
config. For more information, see:

coreos/fedora-coreos-tracker#94 (comment)

The rdcore code supports RAID and LUKS devices, though the latter
needs a new Clevis release with the following patches to be fully
supported:

latchset/clevis#211
latchset/clevis#217

This also implements the root=UUID=$uuid inject patch proposed in
coreos/fedora-coreos-tracker#465.

On its own, this unlocks reprovisioning FCOS with root on a RAID device,
or e.g. in-place reprovisioning of root on btrfs.

Closes: coreos/fedora-coreos-tracker#465
Closes: coreos/fedora-coreos-tracker#94

jlebon · 2020-06-29T21:25:46Z

Requires #184, #354, and the SELinux patch.

With this, it's now possible to configure FCOS with root on a RAID device!

Example Ignition config:

{
  "ignition": {
    "version": "3.2.0-experimental"
  },
  "storage": {
    "disks": [
      {
        "device": "/dev/disk/by-id/virtio-disk1",
        "partitions": [
          {
            "label": "foo"
          }
        ],
        "wipeTable": true
      },
      {
        "device": "/dev/disk/by-id/virtio-disk2",
        "partitions": [
          {
            "label": "bar"
          }
        ],
        "wipeTable": true
      }
    ],
    "raid": [
      {
        "devices": [
          "/dev/disk/by-partlabel/foo",
          "/dev/disk/by-partlabel/bar"
        ],
        "level": "raid1",
        "name": "myroot"
      }
    ],
    "filesystems": [
      {
        "device": "/dev/md/myroot",
        "format": "xfs",
        "wipeFilesystem": true,
        "label": "root"
      }
    ]
  }
}

Running with cosa:

(cosa)$ kola qemuexec -i config.ign --ignition-direct --memory 4096 --add-disk 5G --add-disk 5G
(vm)$ findmnt /sysroot
TARGET   SOURCE     FSTYPE OPTIONS
/sysroot /dev/md127 xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize
(vm)$ lsblk --inverse /dev/md127
NAME    MAJ:MIN RM SIZE RO TYPE  MOUNTPOINT
md127     9:127  0   5G  0 raid1 /sysroot
├─vda1  252:1    0   5G  0 part
│ └─vda 252:0    0   5G  0 disk
└─vdb1  252:17   0   5G  0 part
  └─vdb 252:16   0   5G  0 disk
(vm)$ reboot
...
(vm)$ lsblk --inverse /dev/md127
NAME    MAJ:MIN RM SIZE RO TYPE  MOUNTPOINT
md127     9:127  0   5G  0 raid1 /sysroot
├─vda1  252:1    0   5G  0 part
│ └─vda 252:0    0   5G  0 disk
└─vdb1  252:17   0   5G  0 part
  └─vdb 252:16   0   5G  0 disk
(vm)$ cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt1)/ostree/fedora-coreos-cb9be02cb0885ea378b9763a62c5ca5a596b280d182553fb4f2522725f4257be/vmlinuz-5.6.19-300.jl.fc32.x86_64 mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.md.uuid=139abc1a:9fde35c5:1021a2b6:748ce991 root=UUID=ff2ac1ab-f449c-4de0-8396-b19305747456 rw

jlebon · 2020-06-29T21:32:04Z

For fun, with coreos/ignition#1010, it's possible to do root on RAID10:

{
  "ignition": {
    "version": "3.2.0-experimental"
  },
  "storage": {
    "disks": [
      {
        "device": "/dev/disk/by-id/virtio-disk1",
        "partitions": [
          {
            "label": "foo"
          }
        ],
        "wipeTable": true
      },
      {
        "device": "/dev/disk/by-id/virtio-disk2",
        "partitions": [
          {
            "label": "bar"
          }
        ],
        "wipeTable": true
      },
      {
        "device": "/dev/disk/by-id/virtio-disk3",
        "partitions": [
          {
            "label": "baz"
          }
        ],
        "wipeTable": true
      },
      {
        "device": "/dev/disk/by-id/virtio-disk4",
        "partitions": [
          {
            "label": "boo"
          }
        ],
        "wipeTable": true
      }
    ],
    "raid": [
      {
        "devices": [
          "/dev/disk/by-partlabel/foo",
          "/dev/disk/by-partlabel/bar"
        ],
        "level": "raid1",
        "name": "foobar"
      },
      {
        "devices": [
          "/dev/disk/by-partlabel/baz",
          "/dev/disk/by-partlabel/boo"
        ],
        "level": "raid1",
        "name": "bazboo"
      },
      {
        "devices": [
          "/dev/md/foobar",
          "/dev/md/bazboo"
        ],
        "level": "raid0",
        "name": "myroot"
      }
    ],
    "filesystems": [
      {
        "device": "/dev/md/myroot",
        "format": "xfs",
        "wipeFilesystem": true,
        "label": "root"
      }
    ]
  }
}

Running with cosa:

(cosa)$ kola qemuexec -i config.ign --ignition-direct --memory 4096 --add-disk 5G --add-disk 5G --add-disk 5G --add-disk 5G
(vm)$ findmnt /sysroot
TARGET   SOURCE     FSTYPE OPTIONS
/sysroot /dev/md125 xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize
(vm)$ reboot
...
(vm)$ lsblk --inverse /dev/md125
NAME      MAJ:MIN RM SIZE RO TYPE  MOUNTPOINT
md125       9:125  0  10G  0 raid0 /sysroot
├─md126     9:126  0   5G  0 raid1
│ ├─vdc1  252:33   0   5G  0 part
│ │ └─vdc 252:32   0   5G  0 disk
│ └─vdd1  252:49   0   5G  0 part
│   └─vdd 252:48   0   5G  0 disk
└─md127     9:127  0   5G  0 raid1
  ├─vda1  252:1    0   5G  0 part
  │ └─vda 252:0    0   5G  0 disk
  └─vdb1  252:17   0   5G  0 part
    └─vdb 252:16   0   5G  0 disk
(vm)$ cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt1)/ostree/fedora-coreos-cb9be02cb0885ea378b9763a62c5ca5a596b280d182553fb4f2522725f4257be/vmlinuz-5.6.19-300.jl.fc32.x86_64 mitigations=auto,nosmt systemd.unified_cgroup_hierarchy=0 console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu ostree=/ostree/boot.1/fedora-coreos/cb9be02cb0885ea378b9763a62c5ca5a596b280d182553fb4f2522725f4257be/0  rd.md.uuid=2bbba6df:428aea5c:763ec8b4:9a2a357b rd.md.uuid=7e1033f7:117cde9d8:8cf8492d:619c679b rd.md.uuid=ac2ed991:60d98db2:7cef9638:773d4426 root=UUUID=1b184d4a-f273-46c7-a359-b84f257f07b8 rw

jlebon · 2020-06-29T21:34:18Z

So... I initially wanted to write this in rpm-ostree so it can be in not-bash, and then call it from the initramfs, but it's hard to beat bash in prototyping speed. And then it ended up being much simpler than expected thanks to lsblk's --inverse switch, which gives us exactly what we need. One thing I was tempted to do was providing the ability to have Rust code in this repo which cosa compiles at build-time, but didn't want to tackle that discussion yet.

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap.service

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-uuid-boot.service

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-dracut-rootfs.sh

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap.service

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap

jlebon · 2020-07-02T15:50:30Z

Re. root-on-LUKS, see coreos/ignition#960 (comment).

jlebon · 2020-07-02T19:57:31Z

OK, rebased this and now with #466, #184, and #354 folded in!

Will see how quickly we can get a respin of Clevis with latchset/clevis#211 included. Otherwise, we should be able to hack around it for now by carrying our own version of the path and service units until that happens.

I haven't addressed the comments yet. I'd also like to write a kola test for this before everything goes in. Also need to adapt growpart.

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-growpart

...ay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-rootfs-detect.service

jlebon · 2020-07-07T14:42:30Z

This is blocked on https://bugzilla.redhat.com/show_bug.cgi?id=1845210.

manifest-lock.x86_64.json

cgwalters · 2020-07-12T16:59:08Z

I checked out this branch and also dropped in a locally built kernel from this commit, then provided:

# Example of reprovisioning / with btrfs
variant: fcos
version: 1.0.0
storage:
  filesystems:
    - device: /dev/disk/by-partlabel/root
      format: btrfs
      wipe_filesystem: true
      label: root

And it just worked, rootfs on btrfs was that easy! (And same for ext4 etc.)

(I did get a failure from ignition-ostree-uuid-boot.service but that's #503 (comment) I believe)

cgwalters · 2020-07-13T15:16:10Z

Another bug I just hit with this is when doing an install without providing an Ignition config:

[root@localhost ~]# systemctl status ignition-ostree-rootfs-detect
● ignition-ostree-rootfs-detect.service
     Loaded: not-found (Reason: Unit ignition-ostree-rootfs-detect.service not found.)
     Active: failed (Result: exit-code) since Mon 2020-07-13 15:08:54 UTC; 3min 16s ago
   Main PID: 537 (code=exited, status=1/FAILURE)
        CPU: 2ms

Jul 13 15:08:54 localhost systemd[1]: Starting Ignition OSTree: detect rootfs replacement...
Jul 13 15:08:54 localhost systemd[1]: ignition-ostree-rootfs-detect.service: Main process exited, code=exited, status=1/FAILURE
Jul 13 15:08:54 localhost ignition-ostree-dracut-rootfs[539]: /usr/libexec/ignition-ostree-dracut-rootfs: line 12: /run/ignition.json: No such file or directory
Jul 13 15:08:54 localhost systemd[1]: ignition-ostree-rootfs-detect.service: Failed with result 'exit-code'.
Jul 13 15:08:54 localhost systemd[1]: Failed to start Ignition OSTree: detect rootfs replacement.

The `platform.Conf` type allows abstracting over the different Ignition versions so that different tests can use different versions. Using it instead of the Ignition type directly means that we can now use the 3.2-experimental spec in external tests. This is needed for testing the new LUKS support in Ignition[1] and the related rootfs-on-complex-devices work[2]. [1] coreos/ignition#960 [2] coreos/fedora-coreos-config#503

The `platform.Conf` type allows abstracting over the different Ignition versions so that different tests can use different versions. Using it instead of the Ignition type directly means that we can now use the 3.2-experimental spec in external tests. This is needed for testing the new LUKS support in Ignition[1] and the related rootfs-on-complex-devices work[2]. [1] coreos/ignition#960 [2] coreos/fedora-coreos-config#503 Closes: coreos#1589

arithx · 2020-07-14T05:25:08Z

Another bug I just hit with this is when doing an install without providing an Ignition config:

At least for the no such file part with the new Ignition release the cache config (/run/ignition.json by default) should always exist (coreos/ignition#1002)

cgwalters

This looks good at a high level!

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-growpart

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-dracut-rootfs.sh

jlebon · 2020-08-19T21:36:52Z

OK, so one thing I just realized is that with root=UUID=... always being appended, we lose the prjquota option on subsequent boots because now it's systemd that mounts and not us. And more generally, reprovisioning to a different filesystem with custom mount options will need those persisted as well. So I think rdcore rootmap needs to also inject rootflags=. Working on a patch for that. (Another approach is making it part of /etc/fstab since systemd is capable of re-applying mount options after the initial mount, but we're moving away from /etc/fstab and also some mount options can only be set on the first mount, including quota-related options on XFS).

(Edit: sorry I was kinda confused on this. Mount options for root can't live in the Ignition config. So in the reprovisioning case, it's up to users to add an appropriate rootflags karg.)

This adds basic infrastructure units for "re-provisioning" the root filesystem. See: coreos/fedora-coreos-tracker#94 A unit first detects if the Ignition configuration has a filesystem with the label `root` - if so, we save the rootfs into RAM, let `ignition-disks.service` run, then restore it from RAM. Earlier attempts used the `brd` kernel module which is a RAM-backed block device so we can just `dd`. However, this has some limitations, such as the need to save the full disk in RAM, and the inability for any other initrd code to use `brd` devices. As well, `brd` doesn't support discards, so we require at minimum $rootfs_size RAM (e.g. 3G) until reprovisioning is complete. Future work here will likely move the `restore` phase into `rpm-ostree`. Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>

This is a general best practice; the intention of filesystem UUIDs is that they're unique. It helps backup systems and the like if we change this. This builds on coreos/coreos-assembler@e3905fd In the future, we may also switch to using these UUIDs for subsequent boots; see: coreos/fedora-coreos-tracker#465

This implements the rootmap functionality that figures out all the dependencies required to find `/sysroot`, and injects them into the BLS config. For more information, see: coreos/fedora-coreos-tracker#94 (comment) The `rdcore` code supports RAID and LUKS devices, though the latter needs a new Clevis release with the following patches to be fully supported: latchset/clevis#211 latchset/clevis#217 This also implements the `root=UUID=$uuid` inject patch proposed in coreos/fedora-coreos-tracker#465. On its own, this unlocks reprovisioning FCOS with root on a RAID device, or e.g. in-place reprovisioning of root on btrfs. Closes: coreos/fedora-coreos-tracker#465 Closes: coreos/fedora-coreos-tracker#94

This stamp file was used to make sure coreos-growpart only ran on the first boot when it ran in the real root. Nowadays, it runs in the initramfs as part of `ignition-complete.target` before we even run `ostree-prepare-root` (which meant we were actually creating the stamp file in the initrd filesystem). Having a stamp file is useful though for writing tests. So let's repurpose the idea and put it in `/run` instead.

Add two basic tests: one where we reprovision in place to ext4, and one where we reprovision onto a separate RAID1.

That generator no longer exists.

Split out a small script where the canonical rootflags live in the non-reprovisioning case. This will be used by both `ignition-ostree-mount-sysroot` and `rdcore rootmap`.

jlebon · 2020-08-20T21:38:05Z

OK, so one thing I just realized is that with root=UUID=... always being appended, we lose the prjquota option on subsequent boots because now it's systemd that mounts and not us.

This is fixed now in coreos/coreos-installer#358 and the latest commit here which adds coreos-rootflags. (Definitely a downside of the rdcore approach is that it's more work to get fixes in vs just some script in the overlay.)

jlebon · 2020-08-27T13:08:40Z

--- PASS: ext.fedora-coreos-config_PR-503-4MMBZDZR33QM3J24KCIOGR3JPDUXCUHF5DYIZB4C3WVNVMA5OEJA.root-reprovision.raid1 (95.30s)
...
--- PASS: ext.fedora-coreos-config_PR-503-4MMBZDZR33QM3J24KCIOGR3JPDUXCUHF5DYIZB4C3WVNVMA5OEJA.root-reprovision.filesystem-only (80.73s)

🎉

Who wants to give an approving review on this so we can merge it in?

ashcrow · 2020-08-27T15:48:37Z

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-mount-sysroot.sh

-fi
-mount -o "${mountflags}" "${rootpath}" /sysroot
+
+mount -o "$(coreos-rootflags)" "${rootpath}" /sysroot


This took a second to realize it was sensing options based on a script but 👍!

cgwalters

LGTM

ashcrow

Great work here! I seriously appreciate the additional commenting through the code 🙏

Conan-Kudo · 2020-08-27T16:00:22Z

@jlebon Thanks for making this happen! 🎉 🙏

fulminemizzega · 2020-12-06T03:01:22Z

(Edit: sorry I was kinda confused on this. Mount options for root can't live in the Ignition config. So in the reprovisioning case, it's up to users to add an appropriate rootflags karg.)

Hello,
I think that this should be mentioned in the related documentation section "Reconfiguring the root filesystem". After reading this PR and issue #94 I think I understand why filesystems.mount_options in ignition does not do anything for root fs, but it was not that intuitive (at least to me). In my case, I changed the root fs to btrfs and enabled compression. I also understand this is at least frowned upon, so I guess I got what I deserved.

jlebon · 2020-12-07T14:52:32Z

Hmm, I think if we have magic for reprovisioning root, then we should also just handle mount_options too and translate that into rootflags. This is similar to how with_mount_unit also injects the mount options into the unit.

This was referenced Jun 29, 2020

Add general ignition-ostree module with rootfs replacement #184

Closed

switch to root=<uuid> post-firstboot? coreos/fedora-coreos-tracker#465

Closed

cgwalters reviewed Jun 30, 2020

View reviewed changes

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-inject-rootmap.service Outdated Show resolved Hide resolved

jlebon mentioned this pull request Jun 30, 2020

*: add LUKS coreos/ignition#960

Merged

cgwalters reviewed Jul 1, 2020

View reviewed changes

jlebon force-pushed the pr/rootmap branch from 2b2bf2c to 48afbe5 Compare July 2, 2020 15:43

jlebon mentioned this pull request Jul 2, 2020

dracut/ignition-ostree: Regenerate UUIDs for /boot and / #354

Closed

jlebon force-pushed the pr/rootmap branch from 48afbe5 to 6c3ad4b Compare July 2, 2020 19:52

jlebon mentioned this pull request Jul 2, 2020

*: add Ignition LUKS support #466

Closed

jlebon changed the title ~~WIP: Add coreos-inject-rootmap.service, unlock root-on-RAID~~ WIP: Add coreos-inject-rootmap.service, unlock root-on-{RAID,LUKS} Jul 2, 2020

jlebon changed the title ~~WIP: Add coreos-inject-rootmap.service, unlock root-on-{RAID,LUKS}~~ WIP: Add support for root on complex devices Jul 2, 2020

cgwalters reviewed Jul 2, 2020

View reviewed changes

overlay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/coreos-growpart Outdated Show resolved Hide resolved

jlebon mentioned this pull request Jul 3, 2020

blockdev: support installing to DM devices coreos/coreos-installer#285

Merged

arithx reviewed Jul 7, 2020

View reviewed changes

...ay.d/05core/usr/lib/dracut/modules.d/40ignition-ostree/ignition-ostree-rootfs-detect.service Show resolved Hide resolved

This was referenced Jul 10, 2020

Support for encrypted luks volumes coreos/ignition#577

Closed

Encryption: All disks are belong to us coreos/fedora-coreos-tracker#287

Closed

bgilbert reviewed Jul 10, 2020

View reviewed changes

manifest-lock.x86_64.json Outdated Show resolved Hide resolved

jlebon mentioned this pull request Jul 13, 2020

kola: tweaks around handling of external tests coreos/coreos-assembler#1591

Merged

jlebon force-pushed the pr/rootmap branch from 521ec16 to d48f5da Compare August 11, 2020 19:28

cgwalters reviewed Aug 18, 2020

View reviewed changes

jlebon force-pushed the pr/rootmap branch from d48f5da to c02ddd4 Compare August 18, 2020 21:14

cgwalters and others added 10 commits August 20, 2020 17:19

15coreos-network: add missing word in comment

5461a0a

15coreos-network: drop dep on obsoleted service name

78caf63

40ignition-ostree: skip growpart if moving rootfs

45da6b8

Add root reprovisioning tests

0ba1a3e

Add two basic tests: one where we reprovision in place to ext4, and one where we reprovision onto a separate RAID1.

40ignition-ostree: drop unnecessary inst_simple

46196f9

That generator no longer exists.

40ignition-ostree: split out coreos-rootflags script

faa02d6

Split out a small script where the canonical rootflags live in the non-reprovisioning case. This will be used by both `ignition-ostree-mount-sysroot` and `rdcore rootmap`.

jlebon force-pushed the pr/rootmap branch from c02ddd4 to faa02d6 Compare August 20, 2020 21:24

ashcrow reviewed Aug 27, 2020

View reviewed changes

cgwalters approved these changes Aug 27, 2020

View reviewed changes

ashcrow approved these changes Aug 27, 2020

View reviewed changes

jlebon merged commit f974aff into coreos:testing-devel Aug 27, 2020

jlebon deleted the pr/rootmap branch August 27, 2020 15:55

jlebon mentioned this pull request Aug 27, 2020

Add dedicated page to configuring storage coreos/fedora-coreos-docs#167

Merged

cgwalters mentioned this pull request Sep 8, 2020

Root reprovisioning CI failures coreos/fedora-coreos-tracker#619

Closed

martin-v mentioned this pull request Sep 11, 2020

Example config for encrypted root #614

Closed

bgilbert mentioned this pull request Jan 5, 2021

Copy rootfs Ignition mountOptions to rootflags karg #805

Closed

jlebon mentioned this pull request Jan 14, 2021

coreos-growpart: drop support for growing multipathed disks #816

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

40ignition-ostree: add coreos-inject-rootmap.service #503

40ignition-ostree: add coreos-inject-rootmap.service #503

jlebon commented Jun 29, 2020 •

edited

Loading

jlebon commented Jun 29, 2020

jlebon commented Jun 29, 2020

jlebon commented Jun 29, 2020

jlebon commented Jul 2, 2020

jlebon commented Jul 2, 2020

jlebon commented Jul 7, 2020

cgwalters commented Jul 12, 2020

cgwalters commented Jul 13, 2020

arithx commented Jul 14, 2020 •

edited

Loading

cgwalters left a comment

jlebon commented Aug 19, 2020 •

edited

Loading

jlebon commented Aug 20, 2020

jlebon commented Aug 27, 2020

ashcrow Aug 27, 2020

cgwalters left a comment

ashcrow left a comment

Conan-Kudo commented Aug 27, 2020

fulminemizzega commented Dec 6, 2020

jlebon commented Dec 7, 2020

40ignition-ostree: add coreos-inject-rootmap.service #503

40ignition-ostree: add coreos-inject-rootmap.service #503

Conversation

jlebon commented Jun 29, 2020 • edited Loading

jlebon commented Jun 29, 2020

jlebon commented Jun 29, 2020

jlebon commented Jun 29, 2020

jlebon commented Jul 2, 2020

jlebon commented Jul 2, 2020

jlebon commented Jul 7, 2020

cgwalters commented Jul 12, 2020

cgwalters commented Jul 13, 2020

arithx commented Jul 14, 2020 • edited Loading

cgwalters left a comment

Choose a reason for hiding this comment

jlebon commented Aug 19, 2020 • edited Loading

jlebon commented Aug 20, 2020

jlebon commented Aug 27, 2020

ashcrow Aug 27, 2020

Choose a reason for hiding this comment

cgwalters left a comment

Choose a reason for hiding this comment

ashcrow left a comment

Choose a reason for hiding this comment

Conan-Kudo commented Aug 27, 2020

fulminemizzega commented Dec 6, 2020

jlebon commented Dec 7, 2020

jlebon commented Jun 29, 2020 •

edited

Loading

arithx commented Jul 14, 2020 •

edited

Loading

jlebon commented Aug 19, 2020 •

edited

Loading