cc_disk_setup.py: fix MBR single partition creation #1932

dermotbradley · 2022-12-29T01:34:51Z

Proposed Commit Message

cc_disk_setup.py: fix MBR single partition creation

Fixes the creation of single partitions on MBR devices. Currently this fails with the following debug output:

cc_disk_setup.py[DEBUG]: Calculating partition layout
cc_disk_setup.py[DEBUG]:    Layout is: 0,
cc_disk_setup.py[DEBUG]: Creating partition table on /dev/sdb
subp.py[DEBUG]: Running command ['/sbin/sfdisk', '--Linux', '--unit=S',
'--force', '/dev/sdb'] with allowed return codes [0] (shell=False,
capture=True)
util.py[DEBUG]: Creating partition on /dev/sdb took 0.237 seconds
util.py[WARNING]: Failed partitioning operation
Failed to partition device /dev/sdb
Unexpected error while running command.
Command: ['/sbin/sfdisk/', '--Linux', '--unit=S', '--force', '/dev/sdb']
Exit code: 1
Reason: -
Stdout: Checking that no-one is using this disk right now ... OK

        Disk /dev/sdb: 16 MiB, 16777216 bytes, 32768 sectors
        Disk model: HARDDISK
        Units: sectors of 1 * 512 = 512 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes

        >>> Created a new DOS disklabel with disk identifier 0xb3604c9a.
        /dev/sdb1: Leaving.
Stderr: sfdisk: --Linux option is unnecessary and deprecated
        Start sector 0 out of range.
        Failed to add #1 partition: Result not representable
util.py[DEBUG]: Failed partitioning operation

On a BIOS/MBR partitioned device the 1st partition cannot start at sector 0 as this is reserved for the MBR.

Documentation clarifications/corrections and additional examples added.

Also remove "--Linux" and "--unit=S" options from sfdisk calls, these options have been deprecated since October 2014.

LP: #1851438

Checklist:

My code follows the process laid out in the documentation
I have updated or added any unit tests accordingly
I have updated or added any documentation accordingly

aciba90 · 2023-01-04T19:09:07Z

Test Steps

Execute the following:

IMAGE_NAME=mbr

cat > ./ud.yaml <<EOF
#cloud-config
disk_setup:
  /dev/sdb:
    table_type: mbr
    layout: true
    overwrite: true
fs_setup:
- device: /dev/sdb
  filesystem: ext4
  label: data
  overwrite: true
  partition: auto
mounts:
- ["/dev/sdb1", "/mnt1"]
EOF

lxc delete -f $IMAGE_NAME || true
lxc init --vm ubuntu-daily:lunar $IMAGE_NAME --config=user.user-data="$(cat ./ud.yaml)"

lxc storage volume delete default my-vol || true
lxc storage volume create default my-vol size=1GiB --type=block
lxc config device add ${IMAGE_NAME} my-vol disk pool=default source=my-vol

lxc start $IMAGE_NAME
sleep 5
lxc exec $IMAGE_NAME -- cloud-init status --wait
lxc shell $IMAGE_NAME

Without this fix, cloud-init's log shows:

2023-01-04 19:03:06,410 - handlers.py[DEBUG]: start: init-network/config-disk_setup: running config-disk_setup with frequency once-per-instance
2023-01-04 19:03:06,410 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/1a261e0d-d7d0-4f72-87b6-2d7e8fd43df0/sem/config_disk_setup - wb: [644] 23 bytes
2023-01-04 19:03:06,411 - helpers.py[DEBUG]: Running config-disk_setup using lock (<FileLock using file '/var/lib/cloud/instances/1a261e0d-d7d0-4f72-87b6-2d7e8fd43df0/sem/config_disk_setup'>)
2023-01-04 19:03:06,411 - cc_disk_setup.py[DEBUG]: Partitioning disks: {'/dev/sdb': {'layout': True, 'overwrite': True, 'table_type': 'mbr'}}
2023-01-04 19:03:06,411 - cc_disk_setup.py[DEBUG]: Creating new partition table/disk
2023-01-04 19:03:06,411 - subp.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,429 - cc_disk_setup.py[DEBUG]: Checking values for /dev/sdb definition
2023-01-04 19:03:06,429 - cc_disk_setup.py[DEBUG]: Checking against default devices
2023-01-04 19:03:06,429 - cc_disk_setup.py[DEBUG]: Checking if device /dev/sdb is a valid device
2023-01-04 19:03:06,429 - subp.py[DEBUG]: Running command ['/usr/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,441 - cc_disk_setup.py[DEBUG]: Checking if device layout matches
2023-01-04 19:03:06,441 - subp.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,457 - subp.py[DEBUG]: Running command ['/usr/sbin/partprobe', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,506 - subp.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,545 - subp.py[DEBUG]: Running command ['/usr/sbin/sfdisk', '-l', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,547 - cc_disk_setup.py[DEBUG]: called check_partition_mbr_layout(/dev/sdb, True), returned: []
2023-01-04 19:03:06,547 - cc_disk_setup.py[DEBUG]: Checking if device is safe to partition
2023-01-04 19:03:06,547 - cc_disk_setup.py[DEBUG]: Checking for device size of /dev/sdb
2023-01-04 19:03:06,547 - subp.py[DEBUG]: Running command ['/usr/sbin/blockdev', '--getsize64', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,554 - subp.py[DEBUG]: Running command ['/usr/sbin/blockdev', '--getss', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,555 - cc_disk_setup.py[DEBUG]: Calculating partition layout
2023-01-04 19:03:06,555 - cc_disk_setup.py[DEBUG]:    Layout is: 0,
2023-01-04 19:03:06,555 - cc_disk_setup.py[DEBUG]: Creating partition table on /dev/sdb
2023-01-04 19:03:06,555 - subp.py[DEBUG]: Running command ['/usr/sbin/sfdisk', '--Linux', '--unit=S', '--force', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,565 - util.py[DEBUG]: Creating partition on /dev/sdb took 0.154 seconds
2023-01-04 19:03:06,565 - util.py[WARNING]: Failed partitioning operation
Failed to partition device /dev/sdb
Unexpected error while running command.
Command: ['/usr/sbin/sfdisk', '--Linux', '--unit=S', '--force', '/dev/sdb']
Exit code: 1
Reason: -
Stdout: Checking that no-one is using this disk right now ... OK

        Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
        Disk model: QEMU HARDDISK
        Units: sectors of 1 * 512 = 512 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes

        >>> Created a new DOS (MBR) disklabel with disk identifier 0xac7151aa.
        /dev/sdb1: Leaving.
Stderr: sfdisk: --Linux option is unnecessary and deprecated
        Start sector 0 out of range.
        Failed to add #1 partition: Numerical result out of range
2023-01-04 19:03:06,565 - util.py[DEBUG]: Failed partitioning operation
Failed to partition device /dev/sdb
Unexpected error while running command.
Command: ['/usr/sbin/sfdisk', '--Linux', '--unit=S', '--force', '/dev/sdb']
Exit code: 1
Reason: -
Stdout: Checking that no-one is using this disk right now ... OK

        Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
        Disk model: QEMU HARDDISK
        Units: sectors of 1 * 512 = 512 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes

        >>> Created a new DOS (MBR) disklabel with disk identifier 0xac7151aa.
        /dev/sdb1: Leaving.
Stderr: sfdisk: --Linux option is unnecessary and deprecated
        Start sector 0 out of range.
        Failed to add #1 partition: Numerical result out of range
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 746, in exec_mkpart_mbr
    subp.subp(prt_cmd, data="%s\n" % layout)
  File "/usr/lib/python3/dist-packages/cloudinit/subp.py", line 335, in subp
    raise ProcessExecutionError(
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['/usr/sbin/sfdisk', '--Linux', '--unit=S', '--force', '/dev/sdb']
Exit code: 1
Reason: -
Stdout: Checking that no-one is using this disk right now ... OK

        Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
        Disk model: QEMU HARDDISK
        Units: sectors of 1 * 512 = 512 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes

        >>> Created a new DOS (MBR) disklabel with disk identifier 0xac7151aa.
        /dev/sdb1: Leaving.
Stderr: sfdisk: --Linux option is unnecessary and deprecated
        Start sector 0 out of range.
        Failed to add #1 partition: Numerical result out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 131, in handle
    util.log_time(
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2680, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 872, in mkpart
    exec_mkpart(table_type, device, part_definition)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 792, in exec_mkpart
    return get_dyn_func("exec_mkpart_%s", table_type, device, layout)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 453, in get_dyn_func
    return globals()[func_name](*func_args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_disk_setup.py", line 748, in exec_mkpart_mbr
    raise Exception(
Exception: Failed to partition device /dev/sdb
Unexpected error while running command.
Command: ['/usr/sbin/sfdisk', '--Linux', '--unit=S', '--force', '/dev/sdb']
Exit code: 1
Reason: -
Stdout: Checking that no-one is using this disk right now ... OK

        Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
        Disk model: QEMU HARDDISK
        Units: sectors of 1 * 512 = 512 bytes
        Sector size (logical/physical): 512 bytes / 512 bytes
        I/O size (minimum/optimal): 512 bytes / 512 bytes

        >>> Created a new DOS (MBR) disklabel with disk identifier 0xac7151aa.
        /dev/sdb1: Leaving.
Stderr: sfdisk: --Linux option is unnecessary and deprecated
        Start sector 0 out of range.
        Failed to add #1 partition: Numerical result out of range
2023-01-04 19:03:06,571 - cc_disk_setup.py[DEBUG]: setting up filesystems: [{'device': '/dev/sdb', 'filesystem': 'ext4', 'label': 'data', 'overwrite': True, 'partition': 'auto'}]
2023-01-04 19:03:06,571 - cc_disk_setup.py[DEBUG]: Creating new filesystem.
2023-01-04 19:03:06,571 - subp.py[DEBUG]: Running command ['udevadm', 'settle'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,609 - cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
2023-01-04 19:03:06,609 - cc_disk_setup.py[DEBUG]: Identifying device to create data filesytem on
2023-01-04 19:03:06,609 - subp.py[DEBUG]: Running command ['/usr/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:06,611 - cc_disk_setup.py[DEBUG]: Automatic device for /dev/sdb identified as /dev/sdb
2023-01-04 19:03:06,611 - cc_disk_setup.py[DEBUG]: File system type 'ext4' with label 'data' will be created on /dev/sdb
2023-01-04 19:03:06,611 - cc_disk_setup.py[DEBUG]: Creating file system data on /dev/sdb
2023-01-04 19:03:06,611 - cc_disk_setup.py[DEBUG]:      Using cmd: ['/usr/sbin/mkfs.ext4', '-L', 'data', '-F', '/dev/sdb']
2023-01-04 19:03:06,611 - subp.py[DEBUG]: Running command ['/usr/sbin/mkfs.ext4', '-L', 'data', '-F', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True)
2023-01-04 19:03:07,150 - util.py[DEBUG]: Creating fs for /dev/sdb took 0.579 seconds
2023-01-04 19:03:07,151 - handlers.py[DEBUG]: finish: init-network/config-disk_setup: SUCCESS: config-disk_setup ran successfully

With the fix, cloud-init's log is clean and the disk is correctly partitioned:

$ fdisk -l
...
Disk /dev/sdb: 1 GiB, 1073741824 bytes, 2097152 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x82239156

Device     Boot Start     End Sectors  Size Id Type
/dev/sdb1        2048 2097151 2095104 1023M 83 Linux
...

$ touch /mnt1/asdf.txt

aciba90

Thanks, @dermotbradley, for the improvement.

Aside from the example change, it looks good to me.

cloudinit/config/cc_disk_setup.py

aciba90 · 2023-01-04T19:15:22Z

cloudinit/config/cc_disk_setup.py

-        # Create a single partition
-        return "0,"
+        # Create a single partition, default to Linux
+        return ",,83"


83 / Linux is already the default in sfdisk, so we are not changing behavior in that regard.

I'd added the "83" to align/be consistent with the existing reference on line 631 (to show that "layout: true" and "layout: [100]" are equivalent).

I can change this but it would make sense for the 2 places to be consistent.

Fixes the creation of single partitions on MBR devices. Currently this fails with the following debug output: cc_disk_setup.py[DEBUG]: Calculating partition layout cc_disk_setup.py[DEBUG]: Layout is: 0, cc_disk_setup.py[DEBUG]: Creating partition table on /dev/sdb subp.py[DEBUG]: Running command ['/sbin/sfdisk', '--Linux', '--unit=S', '--force', '/dev/sdb'] with allowed return codes [0] (shell=False, capture=True) util.py[DEBUG]: Creating partition on /dev/sdb took 0.237 seconds util.py[WARNING]: Failed partitioning operation Failed to partition device /dev/sdb Unexpected error while running command. Command: ['/sbin/sfdisk/', '--Linux', '--unit=S', '--force', '/dev/sdb'] Exit code: 1 Reason: - Stdout: Checking that no-one is using this disk right now ... OK Disk /dev/sdb: 16 MiB, 16777216 bytes, 32768 sectors Disk model: HARDDISK Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes >>> Created a new DOS disklabel with disk identifier 0xb3604c9a. /dev/sdb1: Leaving. Stderr: sfdisk: --Linux option is unnecessary and deprecated Start sector 0 out of range. Failed to add canonical#1 partition: Result not representable util.py[DEBUG]: Failed partitioning operation On a BIOS/MBR partitioned device the 1st partition cannot start at sector 0 as this is reserved for the MBR. Fixes LP: #1851438. Documentation clarifications/corrections and additional examples added. Also remove "--Linux" and "--unit=S" options from sfdisk calls, these options have been deprecated since October 2014.

aciba90

LGTM, thanks!

This is technically a change of behavior as the default layout (in the MBR vase) changes from 0 to ,, i.e., from fully spanning the disk to taking as much as possible (respecting the reserved space for MBR).

I understand this is an acceptable change of behavior and I do not need to add a feature flag covering the new behavior as

We are fixing a bug
The difference in size is negligible
I do not see how this could provoke a problem.

I will this open, just in case other maintainers have a different opinion.

aciba90 · 2023-01-05T15:48:15Z

Per https://bugs.launchpad.net/cloud-init/+bug/1851438/comments/4, this issue did start affecting cloud-init since util-linux v2.26 (Xenial).

Thus, all our supported versions fail and we do not change behavior of the existing clients.

aciba90 self-assigned this Jan 4, 2023

aciba90 requested changes Jan 4, 2023

View reviewed changes

dermotbradley force-pushed the disk_setup-mbr-fix branch from 08e9d64 to 73ac605 Compare January 5, 2023 04:47

aciba90 approved these changes Jan 5, 2023

View reviewed changes

aciba90 merged commit 65557c3 into canonical:main Jan 5, 2023

ubuntu-server-builder mentioned this pull request May 12, 2023

cloud-init disk_setup fails to partition disk for Ubuntu18 #3490

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cc_disk_setup.py: fix MBR single partition creation #1932

cc_disk_setup.py: fix MBR single partition creation #1932

dermotbradley commented Dec 29, 2022

aciba90 commented Jan 4, 2023

aciba90 left a comment

aciba90 Jan 4, 2023

dermotbradley Jan 5, 2023

aciba90 left a comment

aciba90 commented Jan 5, 2023 •

edited

cc_disk_setup.py: fix MBR single partition creation #1932

cc_disk_setup.py: fix MBR single partition creation #1932

Conversation

dermotbradley commented Dec 29, 2022

Proposed Commit Message

Checklist:

aciba90 commented Jan 4, 2023

Test Steps

aciba90 left a comment

Choose a reason for hiding this comment

aciba90 Jan 4, 2023

Choose a reason for hiding this comment

dermotbradley Jan 5, 2023

Choose a reason for hiding this comment

aciba90 left a comment

Choose a reason for hiding this comment

aciba90 commented Jan 5, 2023 • edited

aciba90 commented Jan 5, 2023 •

edited