sgdisk: Run partx after partition changes #1717

pothos · 2023-09-29T16:12:08Z

disks: Refuse to modify disks/partitions in use

When a partition or the whole disk is in use, sgdisk should not execute
the destructive operation.
Add a check that errors out when a disk in use or a partition in use is
to be destroyed.
sgdisk: Run partx after partition changes

The sgdisk tool does not update the kernel partition table with BLKPG in
contrast to other similar tools but only uses BLKRRPART which fails as
soon as one partition of the disk is mounted.
Update the kernel partition table with partx when we know that a
partition of the disk is in use.

The sgdisk tool does not update the kernel partition table in contrast to other similar tools. Often udev can detect the changes but not always as experienced when adding a new partition on Flatcar's boot disk. Instead of implicitly relying on some other component to re-read the kernel partition table, trigger the re-read with partprobe. This is proposed in coreos/ignition#1717

This adds coreos/ignition#1717 as downstream patch to fix flatcar/Flatcar#1194

The sgdisk tool does not update the kernel partition table in contrast to other similar tools. Often udev can detect the changes but not always as experienced when adding a new partition on Flatcar's boot disk. Instead of implicitly relying on some other component to re-read the kernel partition table, trigger the re-read with partprobe. This is proposed in coreos/ignition#1717

This adds coreos/ignition#1717 as downstream patch to fix flatcar/Flatcar#1194

The sgdisk tool does not update the kernel partition table in contrast to other similar tools. Often udev can detect the changes but not always as experienced when adding a new partition on Flatcar's boot disk. Instead of implicitly relying on some other component to re-read the kernel partition table, trigger the re-read with partprobe. This is proposed in coreos/ignition#1717

This adds coreos/ignition#1717 as downstream patch to fix flatcar/Flatcar#1194

jlebon

Thanks, this makes sense to me and may actually fix #1729. Some comments but the idea overall LGTM.

docs/release-notes.md

jlebon · 2023-10-16T15:16:15Z

docs/release-notes.md

@@ -15,10 +15,12 @@ nav_order: 9

 ### Changes

+- The Dracut module now installs partprobe


I'd prefer we use blockdev --rereadpt which is part of util-linux and more likely to be already available (e.g. FCOS doesn't ship partprobe).

I tried this and it doesn't achieve the same thing because it can fail with BLKRRPART: Device or resource busy while partprobe succeeds in forcing a kernel partition re-read. If you know that you don't have this problem you could use the introduced command parameter to run blockdev instead, does this work for you?

I guess this requires a wrapper to pass the --rereadpt argument. Your initrd could ship something like that as blockdev-rereadpt and then you configure it as partprobeCmd:

#!/bin/sh exec blockdev --rereadpt "$@"

I tried this and it doesn't achieve the same thing because it can fail with BLKRRPART: Device or resource busy while partprobe succeeds in forcing a kernel partition re-read.

Not sure why it's hitting EBUSY. The common case would be if one of the filesystems on that disk is mounted, but since we were just able to modify the partition table, that shouldn't be the case. Possibly we're racing with udev?

And actually on that topic, would it work here to play the same trick as #1319 and emit a tagged event against the disk?

Right, one of the filesystems is mounted and this happens when adding a new partition to the boot disk.

Ok, so what works is partx -u - /dev/vda and this should have the same effect as partprobe which also adds the individual partitions to the kernel. Since partx should be available everywhere where util-linux is, should I change the PR to use partx?

It seems that sgdisk doesn't do that itself while sfdisk does - is there a big reason to use sgdisk?
Anyway, since it doesn't hurt to call partx I would suggest to use it here and can update the PR.

It seems that sgdisk doesn't do that itself while sfdisk does - is there a big reason to use sgdisk?

I'm not sure. It seems like sfdisk didn't support GPT in the past so that may be related, but I didn't check the timelines. Regardless, changing it at this point is not worth the risk. The implementation is quite tied to sgdisk intricacies.

Anyway, since it doesn't hurt to call partx I would suggest to use it here and can update the PR.

Before we do that, did you try out this suggestion:

And actually on that topic, would it work here to play the same trick as #1319 and emit a tagged event against the disk?

?

That'd be my preferred solution if it works.

I know you're aware, but for posterity: the difference with partx (and partprobe) vs blockdev --rereadpt is that they manually parse the table and tell the kernel about it instead of letting the kernel do it (BLKPG vs BLKRRPART). While there shouldn't be a perceptible difference, I'd rather we keep relying on the kernel and not make the first boot special. (And of course, it avoids another dependency.)

And actually on that topic, would it work here to play the same trick as #1319 and emit a tagged event against the disk?

This here is not a race, it's really because the BLKRRPART ioctl on the block fails when one partition is mounted (E.g., the ESP or root partition). In this case, to my understanding, the partitioning program should add the partition objects one by one with BLKPG (I would argue that normally this is even the preferred way because it avoids the full re-read and async waiting logic needed when wanting to access the created partition).

I haven't tried if supplying sfdisk as sgdisk command would work but anyway, I think we need to work around the deficiencies of sgdisk and run partx/partprobe from Ignition.

udevadm trigger --settle does not result in the new partition showing up, maybe that wasn't clear from my previous comments
(After a reboot, the new partition shows up without problems of course)

internal/sgdisk/sgdisk.go

jlebon · 2023-10-20T16:43:36Z

OK, I think I understand better now.

sgdisk does call BLKRRPART at the end of its operation, but will still exit 0 if it gets EBUSY (and print out its "Warning: The kernel is still using the old partition table." message). So the partition table never gets reloaded. And this happens in your environment because something is keeping one or more filesystems busy on the target disk we're editing.

Now, is that by design (e.g. you're knowingly mounting filesystems before the disks stage runs), or a bug (e.g. something is racing, maybe udev, and probing filesystems between the time sgdisk edits the table and the time it emits BLKRRPART)?

pothos · 2023-10-21T03:54:41Z

Now, is that by design (e.g. you're knowingly mounting filesystems before the disks stage runs), or a bug (e.g. something is racing, maybe udev, and probing filesystems between the time sgdisk edits the table and the time it emits BLKRRPART)?

Correct, it's by design because users want to add a new partition on the boot disk where in our case /usr is already mounted in the initrd, and maybe more (OEM or ESP, I would have to check).

jlebon · 2023-10-25T17:12:58Z

Correct, it's by design because users want to add a new partition on the boot disk where in our case /usr is already mounted in the initrd, and maybe more (OEM or ESP, I would have to check).

Gotcha. And did your initrd change recently to do this mounting earlier than it used to? I'm trying to understand how this ever worked before. I don't think there's a udev rule that will call out to partx/partprobe.

Thinking on this, it's not clear to me whether we should even try to support this (partitioning a busy disk). It makes Ignition less deterministic since depending on the config, it may or may not be able to update the kernel. Though if we can ensure that we correctly error out in the latter case, that'd be OK I guess... I'll try to gather thoughts from a few others.

pothos · 2023-10-30T10:48:13Z

Our initrd uses the /usr partition already at that point. Yes, this was a recent change. By chance, repartitioning the root partition worked because the start point stayed the same even though the "old" partition was still used by the kernel. Adding a new one didn't work anymore which we only found out later.
To me this is really a limitation in sgdisk and with sfdisk this wouldn't occur. I've updated the patch to use partx from util-linux which FCOS probably already has in the initrd (Flatcar did).

jlebon · 2023-11-06T16:59:13Z

I'm OK with this I think if we can confirm that Ignition will correctly error out if we try to nuke/modify partitions that are currently busy (i.e. with mounted filesystems).

In other words, the invariant here is that the loaded partition table must always match the on-disk state at the end of the operation. And fail if that invariant cannot hold.

pothos · 2023-11-07T11:11:42Z

I'm OK with this I think if we can confirm that Ignition will correctly error out if we try to nuke/modify partitions that are currently busy (i.e. with mounted filesystems).

This is orthogonal: currently sgdisk will not error out if you remove or resize a partition in use. Detecting this is also not easy because it's not only mounted filesystems but also about device mapper usage.
This can be used to check for unused disks/partitions:

lsblk -lnpb -o NAME | while IFS= read -r line; do
drive=$(echo "$line" | awk '{print $1}')
mountpoints=$(lsblk -ln -o MOUNTPOINT "$drive")
device_mapper_usages=$(lsblk -ln -o PATH "$drive" |  grep -v "^$drive")
if [[ -z "$mountpoints" ]] && [[ -z "${device_mapper_usages}" ]]; then
  echo "$drive"
fi
done

In other words, the invariant here is that the loaded partition table must always match the on-disk state at the end of the operation. And fail if that invariant cannot hold.

Yes, that's the aim of this PR, as this wasn't upheld before.

pothos · 2023-11-07T11:24:43Z

This is orthogonal: currently sgdisk will not error out if you remove or resize a partition in use. Detecting this is also not easy because it's not only mounted filesystems but also about device mapper usage.

I created a new issue for this: #1745

jlebon · 2023-11-08T23:51:01Z

Let me reword my comment a different way: I think the fact that partitioning a busy disk "additively" worked was a happy accident and not a design choice. No downstream distro until now needed it. This PR is making it a design choice, and I'm OK with that, but then we should try to implement it properly.

I had hoped that partx -u would error out if a busy partition was deleted, but testing it, that's not the case. An strace shows it getting EBUSY for BLKPG_DEL_PARTITION but the command itself doesn't error out.

Re. #1745, I think a simpler way to check for whether the partition is busy is to check if the holders/ directory in sysfs for that device is empty. We do something similar in coreos-installer: https://github.com/coreos/coreos-installer/blob/686e0320f04a5a3e18ab0bb6278a34137eb22f96/src/blockdev.rs#L132

As a bonus, checking this upfront means that we can call partx only when strictly necessary and otherwise rely on BLKRRPART from sgdisk.

Also, it looks like partx is failing in CI on the RAID1 test and some others:

[ 23.100241] ignition[958]: Ignition failed: create partitions failed: re-reading partitions failed: exit status 1: Cmd: "partx" "-u" "-" "/run/ignition/dev_aliases/dev/vda" Stdout: "" Stderr: "partx: /run/ignition/dev_aliases/dev/vda: failed to read partition table\n"

pothos · 2023-11-23T11:28:07Z

When testing I found out that partx -u is not a replacement for partprobe because it does not add new partitions or remove deleted ones. I need separate invocations with partx -d and partx -a with the exact partition numbers to work on and when at it it might make sense to also use the exact partitions for partx -u but that's a minor optimization.

pothos · 2023-12-15T17:21:27Z

The Go 1.21 linter suddenly can't find certain fields anymore, not sure where this comes from. Looks unrelated.

pothos · 2023-12-15T21:41:55Z

I'm still testing this, and will say when it's ready for review

pothos · 2023-12-18T17:26:56Z

I'm still testing this, and will say when it's ready for review

It's ready for review. I've tested it with Flatcar's kola suite and also manually: adding a partition on the boot device, updating a partition (resize), and deleting, and also that modifications of partitions in-use is producing an error and won't be touching the data.

prestist · 2024-01-08T15:45:08Z

I'm still testing this, and will say when it's ready for review

It's ready for review. I've tested it with Flatcar's kola suite and also manually: adding a partition on the boot device, updating a partition (resize), and deleting, and also that modifications of partitions in-use is producing an error and won't be touching the data.

Thank you so much for going through the testing. I will try and take a look today.

When you can, please make sure to re-base, as some of our CI has been updated , and will prevent a merge without it.

Thank you again for your patience and diligence.

pothos · 2024-01-08T16:17:22Z

Thanks for the note, I've rebased it.

prestist

Generally LGTM, I have a few asks and nits if you would not mind taking a look.

Nothing is blocking minus release notes.

docs/release-notes.md

dracut/30ignition/module-setup.sh

internal/distro/distro.go

internal/exec/stages/disks/partitions.go

jlebon · 2024-01-09T19:02:01Z

I've added a review request from myself as well. I should be able to review it soon too.

jlebon

Some minor details, but this looks sane overall. Thanks!

jlebon · 2024-01-12T20:34:31Z

internal/exec/stages/disks/partitions.go

+		return false, fmt.Errorf("failed to resolve %q: %v", blockDev, err)
+	}
+
+	mounts, err := os.Open("/proc/mounts")


A bit heavy to parse /proc/mounts on every block device. Ideally, we'd do this once. In practice, I don't think it matters too much.

Yes, there shouldn't be millions of mount points in the initrd - if that ever is the case it can be cached.

internal/exec/stages/disks/partitions.go

docs/release-notes.md

prestist · 2024-02-27T22:09:27Z

@pothos wanted to see if you had the chance to see @jlebon's change requests?

pothos · 2024-02-29T16:00:45Z

I'll try to have a look next week (edit: in the next months…).

When a partition or the whole disk is in use, sgdisk should not execute the destructive operation. Add a check that errors out when a disk in use or a partition in use is to be destroyed.

The sgdisk tool does not update the kernel partition table with BLKPG in contrast to other similar tools but only uses BLKRRPART which fails as soon as one partition of the disk is mounted. Update the kernel partition table with partx when we know that a partition of the disk is in use.

pothos · 2024-05-07T12:03:43Z

The patches work well in Flatcar PR where a backported them to replace the first patch posted here.

Tested three cases, the overwriting of USR-A was detected and an error thrown, the other two changes passed without problem because the USR-B is unused and can be overwritten and the new partition is also ok to add (confirmed with lsblk that the kernel has the updated values thanks to partx).

variant: flatcar
version: 1.0.0
storage:
  disks:
  - device: /dev/vda
    partitions:
    - label: USR-A
      number: 3
      size_mib: 1
      resize: true

variant: flatcar
version: 1.0.0
storage:
  disks:
  - device: /dev/vda
    partitions:
    - label: USR-B
      number: 4
      size_mib: 1
      resize: true

variant: flatcar
version: 1.0.0
storage:
  disks:
  - device: /dev/vda
    partitions:
    - label: VAR
      number: 10
      size_mib: 1
  filesystems:
  - device: /dev/disk/by-partlabel/VAR
    format: vfat
    path: /var
    label: VAR
    with_mount_unit: true

prestist · 2024-05-17T13:35:23Z

@pothos Thank you so much for testing the cases and addressing @jlebon 's comments. I am going to take a pass at this today and see if I notice anything.

prestist

This looks good to me, thank you again for being so diligent. @jlebon wdyt?

jlebon

A few comments, but looks great overall!

jlebon · 2024-05-22T03:22:39Z

internal/exec/stages/disks/partitions.go

+}
+
+// Expects a /dev/xyz path
+func partitionNumberPrefix(blockDevResolved string) (string, error) {


There is no error path in this function so we could also simplify this to just return a string.

jlebon · 2024-05-22T03:35:50Z

internal/exec/stages/disks/partitions.go

+	scanner := bufio.NewScanner(mounts)
+	for scanner.Scan() {
+		mountSource := strings.Split(scanner.Text(), " ")[0]
+		if strings.Contains(mountSource, "/") {


Minor: likely functionally equivalent, but it'd be clearer/more self-explanatory if instead we did strings.HasPrefix(mountSource, "/").

jlebon · 2024-05-22T03:38:59Z

internal/exec/stages/disks/partitions.go

+func blockDevMounted(blockDevResolved string) (bool, error) {
+	mounts, err := os.Open("/proc/mounts")
+	if err != nil {
+		return false, fmt.Errorf("failed to open mounts: %v", err)


Minor: maybe "failed to open /proc/mounts"?

jlebon · 2024-05-22T17:59:04Z

internal/exec/stages/disks/partitions.go

+	}
+	var partitions []string
+	for _, entry := range entries {
+		if strings.HasPrefix(entry.Name(), blockDevNode) {


Hmm, is that the canonical way to get disk partitions? I wonder how libblkid does it. Which actually, we do link to it in Ignition.

So another way to get this info is to use e.g. getPartitionMap() in this file which calls out to libblkid. But I think that function assumes that the passed device is a disk so it wouldn't work when recursing into the partition in blockDevInUse(). But we could also restructure blockDevInUse() so that the held and mounted checks are their own functions and then not recurse (which anyway feels a bit like a weird thing to do to avoid even looking for partitions on partitions).

Not a blocker.

jlebon · 2024-05-22T18:06:03Z

internal/exec/stages/disks/partitions.go

+	}
+	var activePartitions []string
+	for _, partition := range partitions {
+		partInUse, _, err := blockDevInUse(partition)


At the very least here, we could pass a bool to make it not even check for partitions when recursing here.

prestist · 2024-06-18T15:13:35Z

@pothos have you had a chance to see @jlebon's comments?

pothos mentioned this pull request Oct 2, 2023

ignition: Add partprobe to initrd flatcar/bootengine#73

Merged

pothos marked this pull request as ready for review October 2, 2023 13:39

pothos force-pushed the kai/partprobe branch 2 times, most recently from 8b75dab to 78efdc2 Compare October 2, 2023 13:45

pothos added a commit to flatcar/scripts that referenced this pull request Oct 2, 2023

Run partprobe after Ignition partition changes

eee657b

This adds coreos/ignition#1717 as downstream patch to fix flatcar/Flatcar#1194

pothos mentioned this pull request Oct 2, 2023

Run partprobe after Ignition partition changes flatcar/scripts#1202

Merged

2 tasks

pothos added a commit to flatcar/scripts that referenced this pull request Oct 2, 2023

Run partprobe after Ignition partition changes

2acaf32

This adds coreos/ignition#1717 as downstream patch to fix flatcar/Flatcar#1194

pothos added a commit to flatcar/scripts that referenced this pull request Oct 2, 2023

Run partprobe after Ignition partition changes

7b96fa2

This adds coreos/ignition#1717 as downstream patch to fix flatcar/Flatcar#1194

pothos added a commit to flatcar/scripts that referenced this pull request Oct 2, 2023

Run partprobe after Ignition partition changes

cdf7db4

This adds coreos/ignition#1717 as downstream patch to fix flatcar/Flatcar#1194

pothos added a commit to flatcar/scripts that referenced this pull request Oct 4, 2023

Run partprobe after Ignition partition changes

00bd16b

This adds coreos/ignition#1717 as downstream patch to fix flatcar/Flatcar#1194

pothos mentioned this pull request Oct 4, 2023

Run partprobe after Ignition partition changes flatcar/scripts#1208

Merged

jlebon reviewed Oct 16, 2023

View reviewed changes

jlebon mentioned this pull request Oct 16, 2023

Swap fails to mount due to a race condition #1729

Open

pothos force-pushed the kai/partprobe branch from 8d2f532 to d6f331d Compare October 19, 2023 12:13

pothos force-pushed the kai/partprobe branch from d6f331d to 204e09f Compare October 30, 2023 10:41

pothos changed the title ~~sgdisk: Run partprobe after partition changes~~ sgdisk: Run partx after partition changes Oct 30, 2023

pothos force-pushed the kai/partprobe branch from 204e09f to d637315 Compare October 30, 2023 10:42

pothos force-pushed the kai/partprobe branch from d637315 to 0469e9d Compare November 7, 2023 11:13

pothos mentioned this pull request Nov 7, 2023

Prevent modification of partitions/devices in use #1745

Open

pothos force-pushed the kai/partprobe branch from 0a057b3 to aea76a2 Compare December 15, 2023 17:12

pothos force-pushed the kai/partprobe branch from aea76a2 to 3a4cab0 Compare December 18, 2023 11:40

pothos force-pushed the kai/partprobe branch from 3a4cab0 to 6741b44 Compare January 8, 2024 16:14

prestist reviewed Jan 9, 2024

View reviewed changes

jlebon self-requested a review January 9, 2024 19:01

pothos force-pushed the kai/partprobe branch 2 times, most recently from 1b80a4c to fb2c16a Compare January 12, 2024 13:38

jlebon reviewed Jan 12, 2024

View reviewed changes

pothos force-pushed the kai/partprobe branch 3 times, most recently from 338169e to 0c8f970 Compare May 6, 2024 12:46

pothos added 2 commits May 6, 2024 22:00

disks: Refuse to modify disks/partitions in use

4b5d709

When a partition or the whole disk is in use, sgdisk should not execute the destructive operation. Add a check that errors out when a disk in use or a partition in use is to be destroyed.

pothos force-pushed the kai/partprobe branch from 0c8f970 to be36ed8 Compare May 6, 2024 13:00

pothos requested review from jlebon and prestist May 11, 2024 06:37

prestist approved these changes May 17, 2024

View reviewed changes

jlebon reviewed May 22, 2024

View reviewed changes

		@@ -15,10 +15,12 @@ nav_order: 9

		### Changes

		- The Dracut module now installs partprobe

sgdisk: Run partx after partition changes #1717

Are you sure you want to change the base?

sgdisk: Run partx after partition changes #1717

Conversation

pothos commented Sep 29, 2023 • edited Loading

jlebon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pothos Oct 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pothos Oct 21, 2023 • edited Loading

Choose a reason for hiding this comment

jlebon commented Oct 20, 2023

pothos commented Oct 21, 2023

jlebon commented Oct 25, 2023

pothos commented Oct 30, 2023

jlebon commented Nov 6, 2023

pothos commented Nov 7, 2023

pothos commented Nov 7, 2023

jlebon commented Nov 8, 2023

pothos commented Nov 23, 2023

pothos commented Dec 15, 2023

pothos commented Dec 15, 2023

pothos commented Dec 18, 2023 • edited Loading

prestist commented Jan 8, 2024

pothos commented Jan 8, 2024

prestist left a comment

Choose a reason for hiding this comment

jlebon commented Jan 9, 2024

jlebon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prestist commented Feb 27, 2024

pothos commented Feb 29, 2024 • edited Loading

pothos commented May 7, 2024

prestist commented May 17, 2024

prestist left a comment

Choose a reason for hiding this comment

jlebon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prestist commented Jun 18, 2024

pothos commented Sep 29, 2023 •

edited

Loading

pothos Oct 19, 2023 •

edited

Loading

pothos Oct 21, 2023 •

edited

Loading

pothos commented Dec 18, 2023 •

edited

Loading

pothos commented Feb 29, 2024 •

edited

Loading