[SRU] cloud-init should check/format Azure empheral disks each boot

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1292648

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Precise)', 'cloud-init (Ubuntu Saucy)']
assignee = None
assignee_name = None
date_closed = 2014-08-21T16:46:39.191807+00:00
date_created = 2014-03-14T17:57:43.275829+00:00
date_fix_committed = 2014-03-19T17:49:39.768366+00:00
date_fix_released = 2014-08-21T16:46:39.191807+00:00
id = 1292648
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1292648
milestone = None
owner = smoser
owner_name = Scott Moser
private = False
status = fix_released
submitter = darkmuggle-deactivatedaccount
submitter_name = Ben Howard
tags = ['amd64', 'apport-bug', 'uec-images', 'verification-done']
duplicates = []

Launchpad user Ben Howard(darkmuggle-deactivatedaccount) wrote on 2014-03-14T17:57:43.275829+00:00

SRU Justification

[IMPACT] Users on Windows Azure are guaranteed to have the ephemeral device as ext4 for the first boot. Subsequent boots may result in fuse mounted NTFS file system.

[TEST CASE]
Defined in comment 3.

[Regression Potential] Low. This change is scope only to the Windows Azure datasource and the behavior complies with expected behavior of Windows Azure: the ephemeral disk is not durable between boots. From Microsoft documentation:

Because data on a resource disk may not be durable across reboots, it is often used by
applications and processes running in the virtual machine for transient and temporary
storage of data. It is also used to store page or swap files for the operating system.
(See http://www.windowsazure.com/en-us/documentation/articles/storage-windows-attach-disk/)

Even so, the change to Cloud-init is scope to only replace the ephemeral disk when the disk is 1) NTFS; 2) has a label of "Temporary Storage"; and 3) has no files on it. When the disk matches, cloud-init will turn the code paths for formating the ephemeral disk for that boot only.

[ORIGINAL REPORT]

On Windows Azure, the ephemeral disk should be treated as ephemeral per boot, not per instance.

Microsoft has informed us that under the following conditions an ephemeral disk may disappear:

1. The user resizes the instance
2. A fault causes the instance to move from one physical host to another
3. A machine is shutdown and then started again

Essentially, on Azure, the ephemeral disk is extremely ephemeral. Users who hit any of the above situations are discovering that /mnt is mount with their default NTFS file system.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: cloud-init 0.7.5~bzr964-0ubuntu1 [modified: usr/lib/python2.7/dist-packages/cloudinit/config/cc_disk_setup.py usr/lib/python2.7/dist-packages/cloudinit/config/cc_final_message.py usr/lib/python2.7/dist-packages/cloudinit/config/cc_seed_random.py usr/lib/python2.7/dist-packages/cloudinit/sources/DataSourceAzure.py usr/lib/python2.7/dist-packages/cloudinit/sources/DataSourceCloudSigma.py usr/lib/python2.7/dist-packages/cloudinit/sources/DataSourceSmartOS.py]
ProcVersionSignature: Ubuntu 3.13.0-17.37-generic 3.13.6
Uname: Linux 3.13.0-17-generic x86_64
ApportVersion: 2.13.3-0ubuntu1
Architecture: amd64
Date: Fri Mar 14 17:53:20 2014
PackageArchitecture: all
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

[ubuntu-server-builder]

Launchpad user Ben Howard(darkmuggle-deactivatedaccount) wrote on 2014-03-14T17:57:43.275829+00:00

Launchpad attachments: Dependencies.txt,ProcEnviron.txt

[ubuntu-server-builder]

Launchpad user Ben Howard(darkmuggle-deactivatedaccount) wrote on 2014-03-14T18:10:39.857173+00:00

Uploaded branch that makes the disk formating set the semaphore per device and sets disk formating to "always" for the Azure datasource.

[ubuntu-server-builder]

Launchpad user Ben Howard(darkmuggle-deactivatedaccount) wrote on 2014-03-17T22:56:22.106963+00:00

Test cases:

[ NEW Filesystem created ]

• NTFS file system with label of "Temporary Storage"
  mkfs.ntfs -L "Temporary Storage" -f /dev/sdb1

[ NO New Filesystem created ]

• NTFS file system with label of "Temporary Storage" w/ a file
  mkfs.ntfs -L "Temporary Storage" -f /dev/sdb1; mount /dev/sdb1 /mnt; touch /mnt/test

• NTFS file system, no label
  mkfs.ntfs -f /dev/sdb1

• ext4 file system with label of "Temporary Storage"
  mkfs.ext4 -L "Temporary Storage" /dev/sdb1

• no file system after first boot
  wipefs --all /dev/sdb1

The no file system after first boot should cover the exotic file system and on-disk data formats including LUKS, tar, DRDB and others.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2014-03-19T18:49:15.275510+00:00

This bug was fixed in the package cloud-init - 0.7.5~bzr969-0ubuntu1

---

cloud-init (0.7.5~bzr969-0ubuntu1) trusty; urgency=medium

• New upstream snapshot.
  • Azure: Reformat ephemeral disk if it got re-provisioned
    by the cloud on any reboot (LP: #1292648)
  • final_message: fix replacement of upper case keynames (LP: #1286164)
  • seed_random: do not capture output. Correctly provide
    environment variable RANDOM_SEED_FILE to command.
  • CloudSigma: support base64 encoded user-data
    -- Scott Moser smoser@ubuntu.com Wed, 19 Mar 2014 14:04:34 -0400

[ubuntu-server-builder]

Launchpad user Dave Walker(davewalker) wrote on 2014-03-21T16:44:45.647228+00:00

Hello Ben, or anyone else affected,

Accepted cloud-init into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.6.3-0ubuntu1.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Dave Walker(davewalker) wrote on 2014-03-21T16:46:10.255632+00:00

Hello Ben, or anyone else affected,

Accepted cloud-init into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-init/0.7.3-0ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Ben Howard(darkmuggle-deactivatedaccount) wrote on 2014-03-24T22:01:29.249614+00:00

Performed through validation testing on multiple instances. Marking as verification done.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2014-03-31T06:45:09.120185+00:00

This bug was fixed in the package cloud-init - 0.6.3-0ubuntu1.12

---

cloud-init (0.6.3-0ubuntu1.12) precise-proposed; urgency=low

• debian/patches/lp-1269626-azure_new_instance.patch: fix handling of new
  instances on Windows Azure. Backport of fix from 14.04 (LP: #1269626).
• debian/patches/lp-1292648-azure-format-ephemeral-new.patch: Azure,
  re-format ephemeral disk if necessary (LP: #1292648).
  -- Ben Howard ben.howard@ubuntu.com Tue, 18 Mar 2014 10:58:12 -0600

[ubuntu-server-builder]

Launchpad user Adam Conrad(adconrad) wrote on 2014-03-31T06:45:25.944152+00:00

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2014-03-31T06:45:36.572059+00:00

This bug was fixed in the package cloud-init - 0.7.3-0ubuntu2.2

---

cloud-init (0.7.3-0ubuntu2.2) saucy-proposed; urgency=low

• debian/patches/lp-1269626-azure_new_instance.patch:
  fix handling of new instances on Windows Azure (LP: #1269626).
• debian/patches/lp-1292648-azure-format-ephemeral-new.patch:
  re-format ephemeral disk if necessary (LP: #1292648).
  -- Ben Howard ben.howard@ubuntu.com Wed, 19 Mar 2014 16:31:51 -0600

[ubuntu-server-builder]

Launchpad user Ben Howard(darkmuggle-deactivatedaccount) wrote on 2014-08-21T16:46:56.727045+00:00

This was fixed released a long time ago.

[ubuntu-server-builder]

Launchpad user trialotto(trialotto) wrote on 2015-02-18T04:42:56.481277+00:00

@ben,

Bug returns, the issue is still present in 0.7.5-0ubuntu1.3, when partitioning the ephemeral disk.

Interestingly, a partitioning of form

• [ 66, [33,82]] does return the bug (the whole /dev/sdb becomes ntfs after shutdown). Bug is persistent through resizing.

• [ [ 33,82], 66] does not yield any problem at all. Persisten through resizing (resizing does not matter in this case).

Maybe I did something wrong with the cloud config syntax, but that should not be the case.

Still testing all scenario´s. Any idea of the strange situation?

Kind regards

[ubuntu-server-builder]

Launchpad user Paul Meyer(paul-meyer) wrote on 2016-08-05T16:59:40.001927+00:00

@ben,

I'm seeing a ~100% repro of this bug on the current 16.04 image in Azure on VM resize. Create a D2_v2 VM, check that fstab has /dev/disk/cloud/azure_resource-part1 on /mnt and that /mnt is mounted as ext4. In that case, provisioning went correctly. Now after resizing this VM, e.g. to a D1_v2, /mnt will be mounted as fuseblk with NTFS as the underlying filesystem. The relevant logs from cloud-init:

Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating new filesystem.
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Checking /dev/sdb against default devices
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Manual request of partition 1 for /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Checking device /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/blkid', '-c', '/dev/null', '/dev/sdb1'] with allowed return codes [0, 2] (shell=False, capture=True)
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Device /dev/sdb1 has Temporary Storage ntfs
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Device /dev/sdb1 is cleared for formating
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: File system None will be created on /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] util.py[DEBUG]: Running command ['/bin/lsblk', '--pairs', '--output', 'NAME,TYPE,FSTYPE,LABEL', '/dev/sdb1', '--nodeps'] with allowed return codes [0] (shell=False, capture=True)
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Creating file system None on /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] cc_disk_setup.py[DEBUG]: Using cmd: /sbin/mkfs.ext4 /dev/sdb1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] util.py[DEBUG]: Running command ['/sbin/mkfs.ext4', '/dev/sdb1'] with allowed return codes [0] (shell=False, capture=True)
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: [CLOUDINIT] util.py[DEBUG]: Creating fs for /dev/disk/cloud/azure_resource took 0.038 seconds
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Cloud-init v. 0.7.7 running 'modules:config' at Thu, 04 Aug 2016 18:19:38 +0000. Up 18.80 seconds.
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: 2016-08-04 18:19:38,478 - util.py[WARNING]: Failed during filesystem operation
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Failed to exec of '['/sbin/mkfs.ext4', '/dev/sdb1']':
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Unexpected error while running command.
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Command: ['/sbin/mkfs.ext4', '/dev/sdb1']
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Exit code: 1
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Reason: -
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Stdout: ''
Aug 04 18:19:38 linuxvm-5 cloud-init[2629]: Stderr: 'mke2fs 1.42.13 (17-May-2015)\n/dev/sdb1 is mounted; will not make a filesystem here!\n'

The mnt.mount systemd unit was started before cloud-config.service was started, but that looks by design, looking at the declared dependencies/ordeing in systemd config (cloud-config.service -> cloud-config.target -> cloud-init-local.service -> local-fs.target -> *.mount).

I can open a new bug, but this looks like a regression of this one. Let me know if you want me to open a new one instead.

Thanks, Paul