check_partition_layout has false positives when partitioned with gpt #2897

ubuntu-server-builder · 2023-05-11T01:54:33Z

This bug was originally filed in Launchpad as LP: #1692087

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)', 'cloud-init (Ubuntu Yakkety)', 'cloud-init (Ubuntu Zesty)']
assignee = None
assignee_name = None
date_closed = 2017-09-23T02:33:04.681024+00:00
date_created = 2017-05-19T18:54:46.694358+00:00
date_fix_committed = 2017-05-23T18:12:33.628636+00:00
date_fix_released = 2017-09-23T02:33:04.681024+00:00
id = 1692087
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1692087
milestone = None
owner = smoser
owner_name = Scott Moser
private = False
status = fix_released
submitter = smoser
submitter_name = Scott Moser
tags = ['verification-done-xenial', 'verification-done-yakkety', 'verification-done-zesty']
duplicates = []

Launchpad user Scott Moser(smoser) wrote on 2017-05-19T18:54:46.694358+00:00

=== Begin SRU Template ===
[Impact]
The disk setup module (disk_setup) of cloud-init had several issues
when dealing with GPT formatted devices. The result was that the user
didn't get expected behavior if they were requesting GPT disk labels
or if the disk present already had a GPT label.

[Test Case]
1.) launch an instance with a second disk "/dev/vdb".
This can be done on openstack or azure.

2.) write a config to system.
   $ sudo tee /etc/cloud/cloud.cfg.d/disk-setup.cfg <EOF
#cloud-config
disk_setup:
  /dev/vdb:
    table_type: gpt
    layout: [66, [33, 82]]
    overwrite: True
fs_setup:
- device: /dev/vdb
   partition: 1
   filesystem: ext4
- device: /dev/vdb
   partition: 2
   filesystem: swap
mounts:
    - ["/dev/vdb1", "/mnt"]
EOF

Partition the second disk with a gpt partition table.
   download a paritioning tool and use it.
   $ wget https://git.launchpad.net/~smoser/cloud-init/+git/sru-info/tree/bugs/lp-1686514/disk-setup -O disk-setup
   # format /dev/vdb with GPT disk label and 2 partitions.
   $ sudo LABEL=gpt ./disk-setup ntfs2 /dev/vdb
remove state from the instance and reboot. We expect that the desired
partitioning will be done and the mount written and used on reboot.

$ sudo rm -Rf /var/lib/cloud/ /var/log/cloud-init*
$ sudo reboot

ssh back in and look around
$ grep vdb /proc/mounts
$ grep mnt /etc/fstab

$ sfdisk -l /dev/vdb

sudo reboot
ssh back in and look around.

[Regression Potential]
Regressions will be limited to places where the disk_setup module is used.
That is
a.) Azure (by default on new instances or re-deployed instance)
b.) instances with custom user-data or system configuration.

[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=3507b59eaa4

=== End SRU Template ===

As reported in bug 1686514 (comment 11), there is a bug in the code
that determines if a device layout is "the same" and will thus be re-used.

Even when functional, this code has some serious shortcomings:
a.) it does not check sizes at all, only partition types
b.) it does not check the partition table correctly

The bug to fix here is simply some issues with its usage of sgdisk.
* 'sgdisk -p' was being used to determine the size of a disk.
   this can fail if it believes there is a bad gpt partition table.
* parsing of sgdisk -p output assumed that the 'name' of the partition
   type would not have any spaces (Microsoft basic data)
* interaction with sgdisk did not realize that sgdisk wants input
   of '8300' rather than '83'.

Related bugs:
* bug 1686514: Azure: cloud-init does not handle reformatting GPT partition ephemeral disks
* bug 1691489: fstab entries written by cloud-config may not be mounted

ubuntu-server-builder · 2023-05-11T01:54:35Z

Launchpad user Brian Murray(brian-murray) wrote on 2017-06-13T18:09:07.319978+00:00

Hello Scott, or anyone else affected,

Accepted cloud-init into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ubuntu-server-builder · 2023-05-11T01:54:37Z

Launchpad user Brian Murray(brian-murray) wrote on 2017-06-13T18:56:06.286387+00:00

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ubuntu-server-builder · 2023-05-11T01:54:39Z

Launchpad user Brian Murray(brian-murray) wrote on 2017-06-13T19:09:37.412935+00:00

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-153-g16a7302f-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ubuntu-server-builder · 2023-05-11T01:54:41Z

Launchpad user Joshua Powers(powersj) wrote on 2017-06-20T23:30:09.209617+00:00

Verified on the following releases with versions of cloud-init in proposed:
Zesty (cloud-init all 0.7.9-153-g16a7302f-0ubuntu117.04.1)
Yakkety (0.7.9-153-g16a7302f-0ubuntu116.10.1)
Xenial (0.7.9-153-g16a7302f-0ubuntu1~16.04.1)

In an azure instance with a spare 8Gb disk:

$ az vm create -n $name -g $group --image $image --data-disk-sizes-gb 8 --size Standard_DS2_v2 --ssh-key-value ~/.ssh/id_rsa.pub

$ ls /dev/sd*
/dev/sda /dev/sda1 /dev/sdb /dev/sdb1 /dev/sdc
$ cat /etc/cloud/cloud.cfg.d/disk-setup.cfg
#cloud-config
disk_setup:
/dev/sdc:
table_type: gpt
layout: [66, [33, 82]]
overwrite: True
fs_setup:

device: /dev/sdc
partition: 1
filesystem: ext4
device: /dev/sdc
partition: 2
filesystem: swap
mounts:
- ["/dev/sdc1", "/mnt"]

Reproduce before upgrading:
2017-06-20 22:25:56,571 - cc_mounts.py[WARNING]: Activate mounts: FAIL:mount -a
2017-06-20 22:25:56,572 - util.py[WARNING]: Activate mounts: FAIL:mount -a
Stderr: mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
missing codepage or helper program, or other error

           In some cases useful info is found in syslog - try
           dmesg | tail or so.

After upgrade

powersj@cloud-init-test-powersj-1:/var/log$ ls /dev/sd*
/dev/sda /dev/sda1 /dev/sdb /dev/sdb1 /dev/sdc /dev/sdc1 /dev/sdc2
...
Device Start End Sectors Size Type
/dev/sdc1 2048 11075009 11072962 5.3G Linux filesystem
/dev/sdc2 11075584 16777182 5701599 2.7G Linux swap

No error messages.

ubuntu-server-builder · 2023-05-11T01:54:43Z

Launchpad user Launchpad Janitor(janitor) wrote on 2017-06-27T15:51:34.880334+00:00

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~17.04.1

cloud-init (0.7.9-153-g16a7302f-0ubuntu1~17.04.1) zesty-proposed; urgency=medium

New upstream snapshot.
- net: fix reading and rendering addresses in cidr format.
  [Dimitri John Ledkov] (LP: #1689346, #1684349)
- disk_setup: udev settle before attempting partitioning or fs creation.
  (LP: #1692093)
- GCE: Update the attribute used to find instance SSH keys.
  [Daniel Watkins] (LP: #1693582)
- nplan: For bonds, allow dashed or underscore names of keys.
  [Dimitri John Ledkov] (LP: #1690480)
- tests: python2.6: fix unit tests usage of assertNone and format.
- tests: update docstring on test_configured_list_with_none
- fix tools/ds-identify to not write None twice.
- tox/build: do not package depend on style requirements.
- tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
- flake8: move the pinned version of flake8 up to 3.3.0
- tests: Apply workaround for snapd bug in test case. [Joshua Powers]
- RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
- disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
- function spelling & docstring update [Joshua Powers]
- tests: Fix unittest bug in ntp tests. [Joshua Powers]
- tox: move pylint target to 1.7.1
- Fix get_interfaces_by_mac for empty macs (LP: #1692028)
- DigitalOcean: remove routes except for the public interface.
  [Ben Howard] (LP: #1681531.)
- netplan: pass macaddress, when specified, for vlans
  [Dimitri John Ledkov] (LP: #1690388)
- doc: various improvements for the docs on cc_users_groups.
  [Felix Dreissig]
- cc_ntp: write template before installing and add service restart
  [Ryan Harper] (LP: #1645644)
- tests: fix cloudstack unit tests to avoid accessing
  /var/lib/NetworkManager [Lars Kellogg-Stedman]
- tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
- Actually skip warnings when .skip file is present.
  [Chris Brinker] (LP: #1691551)
- netplan: fix netplan render_network_state signature.
  [Dimitri John Ledkov] (LP: #1685944)
- Azure: fix reformatting of ephemeral disks on resize to large types.
  (LP: #1686514)
- make deb: Add devscripts dependency for make deb.
  Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
- openstack: fix log message copy/paste typo in _get_url_settings
  [Lars Kellogg-Stedman]
- unittests: fix unittests run on centos [Joshua Powers]
- Improve detection of snappy to include os-release and kernel cmdline.
  (LP: #1689944)
- Add address to config entry generated by _klibc_to_config_entry.
  [Julien Castets] (LP: #1691135)
- sysconfig: Raise ValueError when multiple default gateways are present.
  [Chad Smith] (LP: #1687485)
- FreeBSD: improvements and fixes for use on Azure
  [Hongjiang Zhang] (LP: #1636345)
- Add unit tests for ds-identify, fix Ec2 bug found.
- fs_setup: if cmd is specified, use shell interpretation.
  [Paul Meyer] (LP: #1687712)
- doc: document network configuration defaults policy and formats.
  [Ryan Harper]
- doc: Fix name of "uri" key in docs for "cc_apt_configure" module
  [Felix Dreissig]
- tests: Enable artful in integration tests [Joshua Powers]

-- Scott Moser smoser@ubuntu.com Fri, 26 May 2017 16:14:09 -0400

ubuntu-server-builder · 2023-05-11T01:54:45Z

Launchpad user Steve Langasek(vorlon) wrote on 2017-06-27T15:52:56.116246+00:00

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

ubuntu-server-builder · 2023-05-11T01:54:47Z

Launchpad user Launchpad Janitor(janitor) wrote on 2017-06-27T15:54:02.062609+00:00

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~16.10.1

cloud-init (0.7.9-153-g16a7302f-0ubuntu1~16.10.1) yakkety-proposed; urgency=medium

New upstream snapshot.
- net: fix reading and rendering addresses in cidr format.
  [Dimitri John Ledkov] (LP: #1689346, #1684349)
- disk_setup: udev settle before attempting partitioning or fs creation.
  (LP: #1692093)
- GCE: Update the attribute used to find instance SSH keys.
  [Daniel Watkins] (LP: #1693582)
- nplan: For bonds, allow dashed or underscore names of keys.
  [Dimitri John Ledkov] (LP: #1690480)
- tests: python2.6: fix unit tests usage of assertNone and format.
- tests: update docstring on test_configured_list_with_none
- fix tools/ds-identify to not write None twice.
- tox/build: do not package depend on style requirements.
- tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
- flake8: move the pinned version of flake8 up to 3.3.0
- tests: Apply workaround for snapd bug in test case. [Joshua Powers]
- RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
- disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
- function spelling & docstring update [Joshua Powers]
- tests: Fix unittest bug in ntp tests. [Joshua Powers]
- tox: move pylint target to 1.7.1
- Fix get_interfaces_by_mac for empty macs (LP: #1692028)
- DigitalOcean: remove routes except for the public interface.
  [Ben Howard] (LP: #1681531.)
- netplan: pass macaddress, when specified, for vlans
  [Dimitri John Ledkov] (LP: #1690388)
- doc: various improvements for the docs on cc_users_groups.
  [Felix Dreissig]
- cc_ntp: write template before installing and add service restart
  [Ryan Harper] (LP: #1645644)
- tests: fix cloudstack unit tests to avoid accessing
  /var/lib/NetworkManager [Lars Kellogg-Stedman]
- tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
- Actually skip warnings when .skip file is present.
  [Chris Brinker] (LP: #1691551)
- netplan: fix netplan render_network_state signature.
  [Dimitri John Ledkov] (LP: #1685944)
- Azure: fix reformatting of ephemeral disks on resize to large types.
  (LP: #1686514)
- make deb: Add devscripts dependency for make deb.
  Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
- openstack: fix log message copy/paste typo in _get_url_settings
  [Lars Kellogg-Stedman]
- unittests: fix unittests run on centos [Joshua Powers]
- Improve detection of snappy to include os-release and kernel cmdline.
  (LP: #1689944)
- Add address to config entry generated by _klibc_to_config_entry.
  [Julien Castets] (LP: #1691135)
- sysconfig: Raise ValueError when multiple default gateways are present.
  [Chad Smith] (LP: #1687485)
- FreeBSD: improvements and fixes for use on Azure
  [Hongjiang Zhang] (LP: #1636345)
- Add unit tests for ds-identify, fix Ec2 bug found.
- fs_setup: if cmd is specified, use shell interpretation.
  [Paul Meyer] (LP: #1687712)
- doc: document network configuration defaults policy and formats.
  [Ryan Harper]
- doc: Fix name of "uri" key in docs for "cc_apt_configure" module
  [Felix Dreissig]
- tests: Enable artful in integration tests [Joshua Powers]

-- Scott Moser smoser@ubuntu.com Fri, 26 May 2017 16:08:21 -0400

ubuntu-server-builder · 2023-05-11T01:54:49Z

Launchpad user Launchpad Janitor(janitor) wrote on 2017-06-27T15:54:45.867129+00:00

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-0ubuntu1~16.04.1

cloud-init (0.7.9-153-g16a7302f-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

New upstream snapshot.
- net: fix reading and rendering addresses in cidr format.
  [Dimitri John Ledkov] (LP: #1689346, #1684349)
- disk_setup: udev settle before attempting partitioning or fs creation.
  (LP: #1692093)
- GCE: Update the attribute used to find instance SSH keys.
  [Daniel Watkins] (LP: #1693582)
- nplan: For bonds, allow dashed or underscore names of keys.
  [Dimitri John Ledkov] (LP: #1690480)
- tests: python2.6: fix unit tests usage of assertNone and format.
- tests: update docstring on test_configured_list_with_none
- fix tools/ds-identify to not write None twice.
- tox/build: do not package depend on style requirements.
- tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
- flake8: move the pinned version of flake8 up to 3.3.0
- tests: Apply workaround for snapd bug in test case. [Joshua Powers]
- RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration. [Andreas Karis]
- disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
- function spelling & docstring update [Joshua Powers]
- tests: Fix unittest bug in ntp tests. [Joshua Powers]
- tox: move pylint target to 1.7.1
- Fix get_interfaces_by_mac for empty macs (LP: #1692028)
- DigitalOcean: remove routes except for the public interface.
  [Ben Howard] (LP: #1681531.)
- netplan: pass macaddress, when specified, for vlans
  [Dimitri John Ledkov] (LP: #1690388)
- doc: various improvements for the docs on cc_users_groups.
  [Felix Dreissig]
- cc_ntp: write template before installing and add service restart
  [Ryan Harper] (LP: #1645644)
- tests: fix cloudstack unit tests to avoid accessing
  /var/lib/NetworkManager [Lars Kellogg-Stedman]
- tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
- Actually skip warnings when .skip file is present.
  [Chris Brinker] (LP: #1691551)
- netplan: fix netplan render_network_state signature.
  [Dimitri John Ledkov] (LP: #1685944)
- Azure: fix reformatting of ephemeral disks on resize to large types.
  (LP: #1686514)
- make deb: Add devscripts dependency for make deb.
  Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
- openstack: fix log message copy/paste typo in _get_url_settings
  [Lars Kellogg-Stedman]
- unittests: fix unittests run on centos [Joshua Powers]
- Improve detection of snappy to include os-release and kernel cmdline.
  (LP: #1689944)
- Add address to config entry generated by _klibc_to_config_entry.
  [Julien Castets] (LP: #1691135)
- sysconfig: Raise ValueError when multiple default gateways are present.
  [Chad Smith] (LP: #1687485)
- FreeBSD: improvements and fixes for use on Azure
  [Hongjiang Zhang] (LP: #1636345)
- Add unit tests for ds-identify, fix Ec2 bug found.
- fs_setup: if cmd is specified, use shell interpretation.
  [Paul Meyer] (LP: #1687712)
- doc: document network configuration defaults policy and formats.
  [Ryan Harper]
- doc: Fix name of "uri" key in docs for "cc_apt_configure" module
  [Felix Dreissig]
- tests: Enable artful in integration tests [Joshua Powers]

-- Scott Moser smoser@ubuntu.com Fri, 26 May 2017 15:58:48 -0400

ubuntu-server-builder · 2023-05-11T01:54:51Z

Launchpad user Scott Moser(smoser) wrote on 2017-09-23T02:33:07.250109+00:00

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

ubuntu-server-builder added the launchpad Migrated from Launchpad label May 11, 2023

ubuntu-server-builder closed this as completed May 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

check_partition_layout has false positives when partitioned with gpt #2897

check_partition_layout has false positives when partitioned with gpt #2897

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

check_partition_layout has false positives when partitioned with gpt #2897

check_partition_layout has false positives when partitioned with gpt #2897

Comments

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

After upgrade

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023

ubuntu-server-builder commented May 11, 2023