Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support defining mount and swap units via credentials #27260

Closed
bmr-cymru opened this issue Apr 13, 2023 · 19 comments · Fixed by #27563 or #28207
Closed

Support defining mount and swap units via credentials #27260

bmr-cymru opened this issue Apr 13, 2023 · 19 comments · Fixed by #27563 or #28207
Assignees
Labels
creds fstab-generator RFE 🎁 Request for Enhancement, i.e. a feature request
Milestone

Comments

@bmr-cymru
Copy link

bmr-cymru commented Apr 13, 2023

Component

other

Is your feature request related to a problem? Please describe

Linux today supports a variety of block and file system snapshot mechanisms (LVM2, btrfs, split mirrors etc.) and it is possible to boot a simple system with just a single (root) file system into a snapshot of the system state using the existing root= and rootflags= kernel command line options.

Being able to boot into a snapshot has application for rolling back unsuccessful updates, system recovery and troubleshooting.

It is not currently directly possible to boot into a snapshot state for more complex file system layouts (for e.g. with separate /var, /home etc.) without first modifying the snapshot content to reference the appropriate block devices or subvolumes in /etc/fstab.

Describe the solution you'd like

The ability to specify mount and swap units directly on the kernel command line. This would allow (in combination with fstab=no) the overriding of the file system mount structure for a given boot of the operating system image.

This allows a complete snapshot of the system state to be booted for e.g. in the case of a failed upgrade that renders the system unbootable.

The user can then roll back to the snapshot state minimizing down time and providing a valuable safety net around updates to the operating system.

The syntax would be something like:

  systemd.mount=what:where:fstype:options

  systemd.swap=what:options

Where a literal ':' in any of the fields is escaped as '::'.

This could be implemented as a new generator or by extending the existing fstab-generator.

Describe alternatives you've considered

It's possible to implement these options in a separate component that provides a new system generator to recognise and act on the unit syntax. This would require users to install additional packages to take advantage of the feature.

It's also possible to mount the snapshot of the root file system and directly modify /etc/fstab however this complicates rollback (since the changes to the fstab must either be reverted, or preserved in a separate block or file system snapshot).

The systemd version you checked that didn't have the feature you are asking for

253

@bmr-cymru bmr-cymru added the RFE 🎁 Request for Enhancement, i.e. a feature request label Apr 13, 2023
@keszybz
Copy link
Member

keszybz commented Apr 27, 2023

We already support autodetection of swap and many other mount points automatically: https://uapi-group.org/specifications/specs/discoverable_partitions_specification/. This doesn't cover all the cases you describe, but at least for swap it's reasonable to never define it in /etc/fstab. Similarly for /home: it would almost always be shared between different snapshots, so it's just fine to autodiscover the partition and mount it automatically.

@keszybz
Copy link
Member

keszybz commented Apr 27, 2023

Though overall, the request is reasonable. I'm sure we'll various uses for such functionality.

This should be implemented as a generator in systemd.

systemd.mount=what:where:fstype:options
systemd.swap=what:options

This syntax seems reasonable. We use : to separate paths in various places.

@poettering
Copy link
Member

so, this makes a ton of sense to me, and i would like to see this too.

i'd suggest implementing this inside of systemd-fstab-generator, which already interprets root= and usr= as well as /etc/fstab.

I'm not convinced escaping via :: is the way to go. We probably need a way to escape spaces too after all, hence something backslash-based sounds more appropriate, since it can cover all characters.

It would probably be wise doing some reasearch how other components from us and the kernel actually do escaping in similar case, and then follow that.

@poettering
Copy link
Member

There's a line in TODO that asks for the same thing btw, and can be removed by any PR implementing this:

* systemd-fstab-generator: support addition mount specifications via kernel
  cmdline. Usecase: invoke a VM, and mount a host homedir into it via
  virtio-fs.

@bmr-cymru
Copy link
Author

Thanks. I'll take a look at the options for escaping. We have a similar need in libdevmapper where we store file system paths in the kernel aux_data word (which can not contain whitespace).

Something that could be handled by systemd-escape may be a good option since it's already easy for administrators to produce the properly escaped strings.

@Conan-Kudo
Copy link
Contributor

Conan-Kudo commented May 4, 2023

This would also solve a problem for me when dealing with mounting multiple non-nested Btrfs subvolumes to boot up properly (which is how Fedora works right now). There's no standard for "discoverable subvolumes" like there are for "discoverable partitions", so we currently have no fstab-less way to bring up the whole system.

@yuwata
Copy link
Member

yuwata commented May 8, 2023

Waiting in #27563.

yuwata added a commit to yuwata/systemd that referenced this issue May 8, 2023
@bluca
Copy link
Member

bluca commented May 8, 2023

Wouldn't it be better to use credentials for this? This is not kernel configuration, it's purely userspace

@Winterhuman
Copy link
Contributor

Winterhuman commented May 8, 2023

Given the existence of root and mount.usr, it's probably better to have both a cmdline and credential method for specifying this, otherwise it may look like to users that mount.usr is special-cased to exist, but no other partition types can be configured

yuwata added a commit to yuwata/systemd that referenced this issue May 8, 2023
@yuwata
Copy link
Member

yuwata commented May 8, 2023

@bluca and @poettering How creds can be loaded from generators?

yuwata added a commit to yuwata/systemd that referenced this issue May 8, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount=what:where:fstype:options
  systemd.swap=what:options

Closes systemd#27260.
yuwata added a commit to yuwata/systemd that referenced this issue May 8, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount=what:where:fstype:options
  systemd.swap=what:options

Closes systemd#27260.
@bluca
Copy link
Member

bluca commented May 8, 2023

We are picking them up from sd-stub and making them available in /.extra/credentials so they should be available to generators too in the initrd - not sure in terms of what internal helper to call, but in theory it should work, and it's intended to afaik

@keszybz
Copy link
Member

keszybz commented May 8, 2023

I think it's fine to also support credentials. But please keep the command line option too. This is much easier to use for people and familiar.

@bluca
Copy link
Member

bluca commented May 8, 2023

Sure, adding that as well is fine. But I think we should emphasize more the credentials approach, especially for programmatic usage.

yuwata added a commit to yuwata/systemd that referenced this issue May 8, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount=what:where:fstype:options
  systemd.swap=what:options

Closes systemd#27260.
yuwata added a commit to yuwata/systemd that referenced this issue May 9, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount=what:where:fstype:options
  systemd.swap=what:options

Closes systemd#27260.
@bmr-cymru
Copy link
Author

@yuwata thanks! This looks like it will do exactly what we need.

yuwata added a commit to yuwata/systemd that referenced this issue May 11, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount=what:where:fstype:options
  systemd.swap=what:options

Closes systemd#27260.
yuwata added a commit to yuwata/systemd that referenced this issue May 11, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount=what:where:fstype:options
  systemd.swap=what:options

Closes systemd#27260.
yuwata added a commit to yuwata/systemd that referenced this issue May 11, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount=what:where:fstype:options
  systemd.swap=what:options

Closes systemd#27260.
yuwata added a commit to yuwata/systemd that referenced this issue May 11, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount=what:where:fstype:options
  systemd.swap=what:options

Closes systemd#27260.
@Winterhuman
Copy link
Contributor

Winterhuman commented May 11, 2023

Continuing from #27260 (comment), once systemd.mount becomes a thing, what will happen with mount.usr? Will it be kept, but marked as deprecated? Or is it considered better to use the dedicated option when working with usr partitions, therefore it won't be marked as deprecated?

yuwata added a commit to yuwata/systemd that referenced this issue May 19, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount-extra=what:where:fstype:options
  systemd.swap-extra=what:options

Closes systemd#27260.
yuwata added a commit to yuwata/systemd that referenced this issue May 19, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount-extra=what:where:fstype:options
  systemd.swap-extra=what:options

Closes systemd#27260.
yuwata added a commit to yuwata/systemd that referenced this issue May 20, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount-extra=what:where:fstype:options
  systemd.swap-extra=what:options

Closes systemd#27260.
yuwata added a commit to yuwata/systemd that referenced this issue May 23, 2023
…line

Now, the following kernel command line options are supported:
  systemd.mount-extra=what:where:fstype:options
  systemd.swap-extra=what:options

Closes systemd#27260.
@poettering
Copy link
Member

Continuing from #27260 (comment), once systemd.mount becomes a thing, what will happen with mount.usr? Will it be kept, but marked as deprecated? Or is it considered better to use the dedicated option when working with usr partitions, therefore it won't be marked as deprecated?

We should keep that, it's interpreted by different code and gets quite different handling. Moreover, there's usrhash and things too, for referencing /usr/ partitions by verity top-level hash, which really suggests we should leave that as is.

@bluca bluca changed the title Support defining mount and swap units on the kernel command line Support defining mount and swap units on the kernel command line and credentials May 26, 2023
@bluca bluca added the creds label May 26, 2023
@bluca bluca added this to the v254 milestone May 26, 2023
@bluca bluca added the rc-blocker 🚧 PRs and Issues tagged this way are blocking the upcoming rc release! label May 26, 2023
@bluca
Copy link
Member

bluca commented May 26, 2023

Sure, adding that as well is fine. But I think we should emphasize more the credentials approach, especially for programmatic usage.

reopening to track creds support

@bluca bluca reopened this May 26, 2023
@keszybz keszybz changed the title Support defining mount and swap units on the kernel command line and credentials Support defining mount and swap units via credentials Jun 24, 2023
poettering added a commit to poettering/systemd that referenced this issue Jun 30, 2023
poettering added a commit to poettering/systemd that referenced this issue Jun 30, 2023
poettering added a commit to poettering/systemd that referenced this issue Jun 30, 2023
poettering added a commit to poettering/systemd that referenced this issue Jun 30, 2023
poettering added a commit to poettering/systemd that referenced this issue Jun 30, 2023
poettering added a commit to poettering/systemd that referenced this issue Jul 4, 2023
poettering added a commit to poettering/systemd that referenced this issue Jul 4, 2023
poettering added a commit to poettering/systemd that referenced this issue Jul 4, 2023
poettering added a commit to poettering/systemd that referenced this issue Jul 4, 2023
@bmr-cymru
Copy link
Author

I'm trying to test the new systemd.mount-extra in Fedora Rawhide with systemd-254~rc3-1.fc39.x86_64 and LVM2 logical volumes and I'm running into some problems.

Firstly, when using command line mount units and nofstab nothing pulls in systemd-remount-fs.service, which leaves the root fs mounted read-only: I think that unit should be pulled in if command line or credentials fstab entries are present. Other than that the basic case of booting a system with the fstab entries on the kernel command line seems to work as expected:

root=/dev/fedora/root ro rd.lvm.lv=fedora/root rd.lvm.lv=fedora/var rd.lvm.lv=fedora/home fstab=no systemd.mount-extra=/dev/fedora/root:/:ext4:rw systemd.mount-extra=/dev/vda2:/boot:ext4:defaults systemd.mount-extra=/dev/fedora/var:/var:ext4:defaults systemd.mount-extra=/dev/fedora/home:/home:ext4:defaults

The system boots normally and file systems are mounted as expected:

# grep fedora /proc/mounts 
/dev/mapper/fedora-root / ext4 ro,seclabel,relatime 0 0
/dev/mapper/fedora-home /home ext4 rw,seclabel,relatime 0 0
/dev/mapper/fedora-var /var ext4 rw,seclabel,relatime 0 0

Attempting to boot into a snapshot of the system state fails unpredictably: in some cases the system appears to boot normally but systemd attempts to unmount the command line mounts after switching root. In other cases the devices for the command line mount units time out. I'm struggling to debug this: the generated unit files seem sane and I can't figure out why the umounts and timeouts are occurring.

The snapshot command line looks like this:

root=/dev/fedora/root-snap ro rd.lvm.lv=fedora/root-snap rd.lvm.lv=fedora/var-snap rd.lvm.lv=fedora/home-snap fstab=no systemd.mount-extra=/dev/fedora/root-snap:/:ext4:rw systemd.mount-extra=/dev/vda2:/boot:ext4:defaults systemd.mount-extra=/dev/fedora/var-snap:/var:ext4:defaults systemd.mount-extra=/dev/fedora/home-snap:/home:ext4:defaults

In the case that file systems are unmounted the logs look like this:

Jul 25 14:49:36 fedora systemd[1]: Mounting sys-kernel-config.mount - Kernel Configuration File System...
Jul 25 14:49:36 fedora systemd[1]: Mounted sys-kernel-config.mount - Kernel Configuration File System.
Jul 25 14:49:36 fedora systemd[1]: Mounting boot.mount - /boot...
Jul 25 14:49:36 fedora kernel: EXT4-fs (vda2): recovery complete
Jul 25 14:49:36 fedora kernel: EXT4-fs (vda2): mounted filesystem cbb2caf6-429f-4a6a-acc4-9a20564c3f58 r/w with ordered data mode. Quota mode: none.
Jul 25 14:49:36 fedora systemd[1]: Mounted boot.mount - /boot.
Jul 25 14:49:37 fedora systemd[1]: var.mount: Directory /var to mount over is not empty, mounting anyway.
Jul 25 14:49:37 fedora systemd[1]: Mounting var.mount - /var...
Jul 25 14:49:37 fedora kernel: EXT4-fs (dm-8): mounted filesystem 0b8645c2-60d4-4d3a-bcdb-eb2355965ac4 r/w with ordered data mode. Quota mode: none.
Jul 25 14:49:37 fedora systemd[1]: Mounted var.mount - /var.
Jul 25 14:49:37 fedora systemd[1]: Starting dracut-pre-mount.service - dracut pre-mount hook...
Jul 25 14:49:37 fedora systemd[1]: Finished dracut-pre-mount.service - dracut pre-mount hook.
Jul 25 14:49:37 fedora systemd[1]: Mounting home.mount - /home...
Jul 25 14:49:37 fedora systemd[1]: Mounting sysroot.mount - /sysroot...
Jul 25 14:49:37 fedora kernel: EXT4-fs (dm-12): mounted filesystem c8c018f9-a3fc-4cab-9f71-700be61ee322 r/w with ordered data mode. Quota mode: none.
Jul 25 14:49:37 fedora systemd[1]: Mounted home.mount - /home.
Jul 25 14:49:37 fedora kernel: EXT4-fs (dm-3): orphan cleanup on readonly fs
Jul 25 14:49:37 fedora kernel: EXT4-fs (dm-3): mounted filesystem 89564248-d7dc-4e98-ba3a-f5901ef1f756 ro with ordered data mode. Quota mode: none.
Jul 25 14:49:37 fedora systemd[1]: Mounted sysroot.mount - /sysroot.
Jul 25 14:49:37 fedora systemd[1]: Starting initrd-parse-etc.service - Mountpoints Configured in the Real Root...
Jul 25 14:49:37 fedora systemd[1]: Finished initrd-parse-etc.service - Mountpoints Configured in the Real Root.
Jul 25 14:49:37 fedora systemd[1]: dracut-mount.service - dracut mount hook was skipped because no trigger condition checks were met.
Jul 25 14:49:37 fedora systemd[1]: dracut-pre-mount.service: Deactivated successfully.
Jul 25 14:49:37 fedora systemd[1]: Stopped dracut-pre-mount.service - dracut pre-mount hook.
Jul 25 14:49:37 fedora systemd[1]: Starting plymouth-switch-root.service - Plymouth switch root service...
Jul 25 14:49:37 fedora systemd[1]: Finished plymouth-switch-root.service - Plymouth switch root service.
Jul 25 14:49:37 fedora systemd[1]: Reached target initrd-switch-root.target - Switch Root.
Jul 25 14:49:37 fedora systemd[1]: Starting initrd-switch-root.service - Switch Root...
Jul 25 14:49:38 fedora kernel: EXT4-fs (vda2): unmounting filesystem cbb2caf6-429f-4a6a-acc4-9a20564c3f58.
Jul 25 14:49:38 fedora kernel: EXT4-fs (dm-8): unmounting filesystem 0b8645c2-60d4-4d3a-bcdb-eb2355965ac4.
Jul 25 14:49:38 fedora kernel: EXT4-fs (dm-12): unmounting filesystem c8c018f9-a3fc-4cab-9f71-700be61ee322.
Jul 25 14:49:38 fedora systemd[1]: Stopped initrd-switch-root.service - Switch Root.
Jul 25 14:49:38 fedora systemd[1]: Set up automount proc-sys-fs-binfmt_misc.automount - Arbitrary Executable File Formats File System Automount Point.
Jul 25 14:49:38 fedora systemd[1]: Stopped target initrd-switch-root.target - Switch Root.
Jul 25 14:49:38 fedora systemd[1]: Mounting dev-hugepages.mount - Huge Pages File System...
Jul 25 14:49:38 fedora systemd[1]: Mounting dev-mqueue.mount - POSIX Message Queue File System...
Jul 25 14:49:38 fedora systemd[1]: Mounting sys-kernel-debug.mount - Kernel Debug File System...
Jul 25 14:49:38 fedora systemd[1]: Mounting sys-kernel-tracing.mount - Kernel Trace File System...
Jul 25 14:49:38 fedora systemd[1]: Stopped plymouth-switch-root.service - Plymouth switch root service.
Jul 25 14:49:38 fedora systemd[1]: Mounted dev-hugepages.mount - Huge Pages File System.
Jul 25 14:49:38 fedora systemd[1]: Mounted dev-mqueue.mount - POSIX Message Queue File System.
Jul 25 14:49:38 fedora systemd[1]: Mounted sys-kernel-debug.mount - Kernel Debug File System.
Jul 25 14:49:38 fedora systemd[1]: Mounted sys-kernel-tracing.mount - Kernel Trace File System.
Jul 25 14:49:38 fedora systemd[1]: Mounting sys-fs-fuse-connections.mount - FUSE Control File System...
Jul 25 14:49:38 fedora systemd[1]: Mounted sys-fs-fuse-connections.mount - FUSE Control File System.
Jul 25 14:49:38 fedora systemd[1]: Mounting home.mount - /home...
Jul 25 14:49:38 fedora systemd[1]: Mounting var.mount - /var...
Jul 25 14:49:38 fedora kernel: EXT4-fs (dm-12): mounted filesystem c8c018f9-a3fc-4cab-9f71-700be61ee322 r/w with ordered data mode. Quota mode: none.
Jul 25 14:49:38 fedora kernel: EXT4-fs (dm-8): mounted filesystem 0b8645c2-60d4-4d3a-bcdb-eb2355965ac4 r/w with ordered data mode. Quota mode: none.
Jul 25 14:49:38 fedora systemd[1]: Mounted home.mount - /home.
Jul 25 14:49:38 fedora systemd[1]: Mounted var.mount - /var.
Jul 25 14:49:38 fedora systemd[1]: tmp.mount: Directory /tmp to mount over is not empty, mounting anyway.
Jul 25 14:49:38 fedora systemd[1]: Mounting tmp.mount - Temporary Directory /tmp...
Jul 25 14:49:38 fedora systemd[1]: Mounted tmp.mount - Temporary Directory /tmp.
Jul 25 14:49:38 fedora systemd[1]: Unmounting home.mount - /home...
Jul 25 14:49:38 fedora kernel: EXT4-fs (dm-12): unmounting filesystem c8c018f9-a3fc-4cab-9f71-700be61ee322.
Jul 25 14:49:38 fedora kernel: EXT4-fs (vda2): mounted filesystem cbb2caf6-429f-4a6a-acc4-9a20564c3f58 r/w with ordered data mode. Quota mode: none.
Jul 25 14:49:38 fedora systemd[1]: Mounting boot.mount - /boot...
Jul 25 14:49:38 fedora systemd[1]: home.mount: Deactivated successfully.
Jul 25 14:49:38 fedora systemd[1]: Unmounted home.mount - /home.
Jul 25 14:49:38 fedora systemd[1]: Mounted boot.mount - /boot.
Jul 25 14:49:38 fedora systemd[1]: Unmounting var.mount - /var...
Jul 25 14:49:38 fedora umount[807]: umount: /var: target is busy.
Jul 25 14:49:38 fedora systemd[1]: var.mount: Mount process exited, code=exited, status=32/n/a
Jul 25 14:49:38 fedora systemd[1]: Failed unmounting var.mount - /var.

(the umount for /var fails since the target is busy but /home is unmounted at the end of booting).

@yuwata
Copy link
Member

yuwata commented Jul 25, 2023

@bmr-cymru Filed as #28516. I will take a look.

@bluca bluca removed the rc-blocker 🚧 PRs and Issues tagged this way are blocking the upcoming rc release! label Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment