base64 user-data broken in OVF datasource

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1619394

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)']
assignee = None
assignee_name = None
date_closed = 2016-09-12T20:52:03.343752+00:00
date_created = 2016-09-01T17:17:27.525158+00:00
date_fix_committed = 2016-09-12T20:52:03.343752+00:00
date_fix_released = 2016-09-12T20:52:03.343752+00:00
id = 1619394
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1619394
milestone = None
owner = jvassev
owner_name = Julian Vassev
private = False
status = fix_released
submitter = jvassev
submitter_name = Julian Vassev
tags = ['verification-done']
duplicates = []

Launchpad user Julian Vassev(jvassev) wrote on 2016-09-01T17:17:27.525158+00:00

==== Begin SRU Template ====
[Impact]
Users of the OVF datasource can not provide user-data that is base64 encoded.

[Test Case]
$ rel=xenial
$ burl="http://cloud-images.ubuntu.com/daily/server/$rel/current/"
$ disk="$rel-server-cloudimg-amd64-disk1.img"
$ wget "$burl/$disk" -O "$disk"

prepare the ovftransport.iso

$ git clone http://git.launchpad.net/cloud-init
$ ( cd cloud-init/doc/sources/ovf && ./make-iso ovf-env.xml.tmpl user-data --output - ) > ovftransport.iso

create a temp disk for patching and patch in -proposed

$ qemu-img create -f qcow2 -b "$disk" disk.img
$ sudo mount-image-callback --system-resolvconf disk.img -- chroot MOUNTPOINT
sh -c 'l=/etc/apt/sources.list.d/proposed.list;
echo deb http://archive.ubuntu.com/ubuntu/ xenial-proposed main > "$l" &&
apt-get -q update && apt-get install -qy cloud-init'

$ qemu-system-x86_64 -enable-kvm
-device virtio-net-pci,netdev=net00 -netdev type=user,id=net00
-hda disk.img -cdrom ovftransport.iso -m 512 -curses -snapshot

you will see an 'ovfdemo' login prompt, log in as root:passw0rd

verify it worked by cat /var/lib/cloud/instance/user-data.txt

[Regression Potential]
very low chance for regression, a guest with user-data would be broken previously.

==== End SRU Template ====

I am using cloudint 0.7.7bzr1256-0ubuntu116.04.1 on Ubuntu 16.04.1 LTS.

When I pass yaml-formatted user-data over the OVF datasource over the ISO transport the yaml string fails to be parsed.

I tracked this to minidom's inability to handle newlines in an attribute. The xml bellow works for coreos but breaks under cloud-init:

         

If I use base64-encoded user-data I get this error:

  Sep 01 12:07:43 sof2-lab3-dhcp371 cloud-init[3248]: 2016-09-01 12:07:43,854 - init.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'I2Nsb3VkLWNvbmZpZwotLS0K'...'
  Sep 01 12:07:43 sof2-lab3-dhcp371 cloud-init[3248]: [CLOUDINIT] init.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'b'I2Nsb3VkLWNvbmZpZwotLS0K'...'

Is there a way to pass user-data as a single-line string that doesn't confuse minidom?

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-09-12T16:35:12.139679+00:00

fwiw, i'm pretty sure it is invalid ovf to have newlines in the value of property.
minidom might be doing the right thing.

the fix for this in cloud-init is to allow you to pass in base64 encoded string in that user-data.

that path was broken in python3.

Julian put a branch to fix that at https://code.launchpad.net/~jvassev/cloud-init/+git/cloud-init/+ref/fix-b64decode-ovf .

I've put a branch up at https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/305496 with a unit test, and i'm going to pull that in now.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-09-12T20:52:02.166907+00:00

fixed in 0.7.8.

[ubuntu-server-builder]

Launchpad user Chris J Arges(arges) wrote on 2016-09-13T20:18:46.234147+00:00

Hello Julian, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Julian Vassev(jvassev) wrote on 2016-09-14T10:41:28.121327+00:00

Hi Chris,
Upgrading to 0.7.7-31-g65ace7b-0ubuntu1~16.04.1 from -proposed fixes my issue.

[ubuntu-server-builder]

Launchpad user Martin Pitt(pitti) wrote on 2016-09-15T14:31:02.597110+00:00

Hello Julian, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-1-g3705bb5-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-09-17T02:06:41.541103+00:00

verified as shown in SRU template.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2016-09-22T17:34:08.065412+00:00

This bug was fixed in the package cloud-init - 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

---

cloud-init (0.7.8-1-g3705bb5-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

• New upstream release 0.7.8.
• New upstream snapshot.
  • systemd: put cloud-init.target After multi-user.target (LP: #1623868)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

• debian/control: add Breaks of older versions of walinuxagent (LP: #1623570)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

• debian/control: fix missing dependency on python3-serial,
  and make SmartOS datasource work.
• debian/cloud-init.templates fix capitalisation in template so
  dpkg-reconfigure works to select OpenStack. (LP: #1575727)
• d/README.source, d/control, d/new-upstream-snapshot, d/rules: sync
  with yakkety for changes due to move to git.
• d/rules: change PYVER=python3 to PYVER=3 to adjust to upstream change.
• debian/rules, debian/cloud-init.install: remove install file
  to ensure expected files are collected into cloud-init deb.
  (LP: #1615745)
• debian/dirs: remove obsolete / unused file.
• upstream move from bzr to git.
• New upstream snapshot.
  • Allow link type of null in network_data.json [Jon Grimm] (LP: #1621968)
  • DataSourceOVF: fix user-data as base64 with python3 (LP: #1619394)
  • remove obsolete .bzrignore
  • systemd: Better support package and upgrade. (LP: #1576692, #1621336)
  • tests: cleanup tempdirs in apt_source tests
  • apt config conversion: treat empty string as not provided. (LP: #1621180)
  • Fix typo in default keys for phone_home [Roland Sommer] (LP: #1607810)
  • salt minion: update default pki directory for newer salt minion.
    (LP: #1609899)
  • bddeb: add --release flag to specify the release in changelog.
  • apt-config: allow both old and new format to be present.
    [Christian Ehrhardt] (LP: #1616831)
  • python2.6: fix dict comprehension usage in _lsb_release. [Joshua Harlow]
  • Add a module that can configure spacewalk. [Joshua Harlow]
  • add install option for openrc [Matthew Thode]
  • Generate a dummy bond name for OpenStack (LP: #1605749)
  • network: fix get_interface_mac for bond slave, read_sys_net for ENOTDIR
  • azure dhclient-hook cleanups
  • Minor cleanups to atomic_helper and add unit tests.
  • Fix Gentoo net config generation [Matthew Thode]
  • distros: fix get_primary_arch method use of os.uname [Andrew Jorgensen]
  • Apt: add new apt configuration format [Christian Ehrhardt]
  • Get Azure endpoint server from DHCP client [Brent Baude]
  • DigitalOcean: use the v1.json endpoint [Ben Howard]
  • MAAS: add vendor-data support (LP: #1612313)
  • Upgrade to a configobj package new enough to work [Joshua Harlow]
  • ConfigDrive: recognize 'tap' as a link type. (LP: #1610784)
  • NoCloud: fix bug providing network-interfaces via meta-data.
    (LP: 1577982)
  • Add distro tags on config modules that should have it [Joshua Harlow]
  • ChangeLog: update changelog for previous commit.
  • add ntp config module [Ryan Harper]
  • SmartOS: more improvements for network configuration
  • tools/read-version: update to address change in version
  • make-tarball: older versions of git with --format=tar.
  • read-version: do not attempt git-describe if no git.
  • Newer requests have strong type validation [Joshua Harlow]
  • For upstream snapshot versions do not modify git-describe output.
  • adjust signal_handler for version changes.
  • revert unintended change to ubuntu sources list
  • drop modification of version during make-tarball, tools changes.
  • adjust tools and version information.
  • Update build tools to work with git [Lars Kellogg-Stedman]
  • fix pep8 errors in mcollective unit tests
  • mcollective: add tests, cleanups and bug fix when no config in /etc.

-- Scott Moser smoser@ubuntu.com Thu, 15 Sep 2016 09:57:27 -0400

[ubuntu-server-builder]

Launchpad user Chris J Arges(arges) wrote on 2016-09-22T17:35:43.941210+00:00

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.