Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SRU] Custom vendor data causes cloud-init failure on 0.7.5 #2565

Closed
ubuntu-server-builder opened this issue May 10, 2023 · 14 comments
Closed

[SRU] Custom vendor data causes cloud-init failure on 0.7.5 #2565

ubuntu-server-builder opened this issue May 10, 2023 · 14 comments
Labels
launchpad Migrated from Launchpad

Comments

@ubuntu-server-builder
Copy link
Collaborator

This bug was originally filed in Launchpad as LP: #1469260

Launchpad details 
affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Trusty)', 'cloud-init (Ubuntu Utopic)']
assignee = None
assignee_name = None
date_closed = 2015-07-10T18:08:30.713706+00:00
date_created = 2015-06-26T18:24:12.953850+00:00
date_fix_committed = 2015-07-10T18:08:30.713706+00:00
date_fix_released = 2015-07-10T18:08:30.713706+00:00
id = 1469260
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1469260
milestone = None
owner = erhudy
owner_name = Edmund Rhudy
private = False
status = fix_released
submitter = erhudy
submitter_name = Edmund Rhudy
tags = ['openstack', 'sts', 'verification-done']
duplicates = []

Launchpad user Edmund Rhudy(erhudy) wrote on 2015-06-26T18:24:12.953850+00:00

[Impact]

When a vendor data json provides a dictionary without a 'cloud-init' key, cloud-init renders a non functional user-data, so any configuration (i.e. ssh public keys to use) is missed.

This prevents cloud providers from publishing a vendor data that is not intended to be consumed by cloud-init.

This patch checks for the existence of 'cloud-init' key and tries to get None, a string or a list as value, if this process fails or cloud-init key is missing the vendor data is set to None.

[Test Case]

  • deploy an OpenStack cloud (easy right? :) )
  • configure vendor data
    • Edit /etc/nova/nova.conf in neutron-gateway unit(s), include the following two lines:
      vendordata_driver=nova.api.metadata.vendordata_json.JsonFileVendorData
      vendordata_jsonfile_path=/etc/nova/vendordata.json
    • Create /etc/nova/vendordata.json in neutron-gateway unit(s) with the following content:
      {"custom": {"a": 1, "b": [2, 3]}}
    • Restart nova-api-metadata (sudo service nova-api-metadata restart)
  • Launch an instance using trusty

Expected result:

  • the new instance is launched and is accesible according to the configuration used

Actual result:

  • cloud-init fails to configure the ssh public key

[Regression Potential]

  • This patch is already part of Vivid and there are no known issues.
  • This proposed fix was tested with a custom image and no issues were detected.

[Other Info]

I encountered this issue when adding custom vendor data via nova-compute. Originally the bug manifested as SSH host key generation failing to fire when vendor data was present (example vendor data below).

{"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}

I launched a volume-backed instance, waited for it to fail, then terminated it and mounted its root volume to examine the logs. What I found was that cloud-init was failing to process vendor-data into MIME multipart (note the absence of the line that indicates that cloud-init is writing vendor-data.txt.i):

2015-06-25 21:41:02,178 - util.py[DEBUG]: Writing to /var/lib/cloud/instance/obj.pkl - wb: [256] 9751 bytes
2015-06-25 21:41:02,178 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/user-data.txt - wb: [384] 0 bytes
2015-06-25 21:41:02,184 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/user-data.txt.i - wb: [384] 345 bytes
2015-06-25 21:41:02,185 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/65c9fb0c-0700-4f87-a22f-c59534e98dfb/vendor-data.txt - wb: [384] 234 bytes
2015-06-25 21:41:02,185 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)

After following the call chain all the way down, I found the problematic code in user_data.py:

Coverts a raw string into a mime message

def convert_string(raw_data, headers=None):
    if not raw_data:
        raw_data = ''
    if not headers:
        headers = {}
    data = util.decomp_gzip(raw_data)
    if "mime-version:" in data[0:4096].lower():
        msg = email.message_from_string(data)
        for (key, val) in headers.iteritems():
            _replace_header(msg, key, val)
    else:
        mtype = headers.get(CONTENT_TYPE, NOT_MULTIPART_TYPE)
        maintype, subtype = mtype.split("/", 1)
        msg = MIMEBase(maintype, subtype, *headers)
        msg.set_payload(data)
    return msg

raw_data in the case that is failing is a dictionary rather than the expected string, so slicing into data causes a TypeError: unhashable type exception.

I think this bug was fixed after a fashion in 0.7.7, where the call to util.decomp_gzip() is now wrapped by util.decode_binary(), which appears to always return a string.
@ubuntu-server-builder ubuntu-server-builder added the launchpad Migrated from Launchpad label May 10, 2023
@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Scott Moser(smoser) wrote on 2015-07-10T18:08:19.431199+00:00

marking trunk fix-released and ubuntu fixed-released per bug openers suggestion that it was fixed in 0.7.7 . I have not reproduced though.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Edmund Rhudy(erhudy) wrote on 2015-07-22T15:46:15.301377+00:00

I can reliably reproduce this issue via the following steps:

  1. Create blah.py under nova/api/metadata/ with these contents:

from nova.api.metadata import base

class CustomVendordata(base.VendorDataDriver):
def init(self, *args, **kwargs):
super(BcpcMetadata, self).init(*args, **kwargs)

def get(self):
    return {"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}
  1. Configure Nova to use that class as the vendordata_driver in nova.conf:

[DEFAULT]
vendordata_driver = nova.api.metadata.blah.CustomVendordata

  1. Restart nova-api.

  2. Launch an Ubuntu cloud instance on that hypervisor.

  3. Experience cloud-init fury:

2015-07-22 15:30:56,075 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
-----END SSH HOST KEY KEYS-----

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Edmund Rhudy(erhudy) wrote on 2015-07-22T16:15:27.272942+00:00

Just doing something simple like

data = str(util.decomp_gzip(raw_data))

causes it to work, because data's now a string and hashable and so the range data[0:4096] works (whether the content is actually what it's supposed to be is another question).

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Edmund Rhudy(erhudy) wrote on 2015-07-23T14:16:41.225726+00:00

Screwed up the above script, it should be

from nova.api.metadata import base

class CustomVendordata(base.VendorDataDriver):
def init(self, *args, **kwargs):
super(CustomVendordata, self).init(*args, **kwargs)

def get(self):
    return {"msg": "", "uuid": "4996e2b67d2941818646481453de1efe", "users": [{"username": "erhudy", "sshPublicKeys": [], "uuid": "erhudy"}], "name": "TestTenant"}

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Edmund Rhudy(erhudy) wrote on 2015-07-23T14:57:01.299847+00:00

Changing the class to return a string rather than a dictionary avoids the cloud-init explosion. That runs counter to all the documentation that says "use JSON", but it's the workaround we'll use for now (serializing to YAML).

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Felipe Reyes(freyes) wrote on 2015-07-23T19:22:52.749881+00:00

A vendor data file should be a dict with a cloud-init key and the value for that key is expected to be a string or a list[0], the docs provide a simple example[1]

Here is another example that will add a user called cloudy, upgrade the system and install htop:

{"cloud-init": "#cloud-config\nusers:\n - name: cloudy\n ssh-import-id: cloudy\npackage_upgrade: True\npackages:\n - htop"}

So a yaml cloud-config compatible has to be passed a string.

[0] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/sources/helpers/openstack.py#L470
[1] http://cloudinit.readthedocs.org/en/latest/topics/datasources.html?highlight=vendor#vendor-data

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Felipe Reyes(freyes) wrote on 2015-07-28T01:24:36.109812+00:00

cloud-init misbehaves when a vendor data json comes with information that won't be consumed by it, a fix was added in rev 1013[0]. I backported this patch to Trusty and prepared a patched image, having a custom vendor data doesn't break cloud-init functionality.

I'll submit a SRU to fix this in Trusty as soon as possible.

[0] http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/1013

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Felipe Reyes(freyes) wrote on 2015-07-28T01:25:09.974689+00:00

Utopic is already EOL, so I'm marking it as Invalid

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Felipe Reyes(freyes) wrote on 2015-07-28T03:18:29.196456+00:00

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Felipe Reyes(freyes) wrote on 2015-09-10T15:54:44.390965+00:00

Rebased patch on top of latest cloud-init available in trusty-updates.
Launchpad attachments: lp1469260_trusty.debdiff

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Chris J Arges(arges) wrote on 2015-09-16T15:47:04.780721+00:00

Hello Edmund, or anyone else affected,

Accepted cloud-init into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.5-0ubuntu1.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Felipe Reyes(freyes) wrote on 2015-09-22T02:43:34.306970+00:00

The package available in trusty-proposed fixes this bug.

$nova console-log 7a2e57a7-df5c-4c61-8f69-c98a41a1de5f
...
ubuntu login: Cloud-init v. 0.7.5 running 'modules:final' at Tue, 22 Sep 2015 02:39:29 +0000. Up 26.16 seconds.
2015-09-22 02:39:30,077 - util.py[WARNING]: Running ssh-authkey-fingerprints (<module 'cloudinit.config.cc_ssh_authkey_fingerprints' from '/usr/lib/python2.7/dist-packages/cloudinit/config/cc_ssh_authkey_fingerprints.pyc'>) failed
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
-----END SSH HOST KEY KEYS-----
Cloud-init v. 0.7.5 finished at Tue, 22 Sep 2015 02:39:30 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 26.33 seconds

$ nova console-log trusty-patched
...
Cloud-init v. 0.7.5 running 'modules:final' at Tue, 22 Sep 2015 02:41:53 +0000. Up 30.54 seconds.
ci-info: ++++++++++Authorized keys from /home/ubuntu/.ssh/authorized_keys for user ubuntu+++++++++++
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ci-info: | Keytype | Fingerprint (md5) | Options | Comment |
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ci-info: | ssh-rsa | bd:37:fc:87:b5:3a:c6:4b:86:cf:ab:e1:29:81:aa:7f | - | Generated-by-Nova |
ci-info: +---------+-------------------------------------------------+---------+-------------------+
ec2:
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 35:8e:8c:4d:86:ff:2c:a0:01:da:f0:fd:db:dc:da:94 root@trusty-patched (DSA)
ec2: 256 72:f7:f1:6e:5d:74:ad:ad:13:25:2a:3f:25:83:8b:5c root@trusty-patched (ECDSA)
ec2: 256 33:cb:4f:2a:70:08:30:8f:dc:3e:60:68:1a:19:a9:cc root@trusty-patched (ED25519)
ec2: 2048 60:cb:95:fe:11:1f:01:14:fc:44:6e:a6:43:1b:81:31 root@trusty-patched (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGp6OpR/cqHVcQmks95ANFCbgi67E+Opr4CmhOIDZl1zRkT6hwW2qGZa3P0Y9PAisNorgaidbpq+Vfi13MKsV5o= root@trusty-patched
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPy5DB2nOuBzAL+R+0x4xPn3bfdGCSlLgEK1B7PIUzyB root@trusty-patched
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCY7NO6k1FBQ27JfcQSuewZ74NNCWhC0ZllDtGi3j/WoQooGn7zRWcY36yibvFu7JP1k9ajrPqUD9P0wlDLhfdSaOzyKWOEqP/pSW2/o1R+o/L7+mOFdZoOblzjd+hHvjkZ+xXMRAn0o/fvRo7kButtKyijscGvUZ8yAFRRNauseJxzsMBBdX0/AWLRNG6LFMjrsWwgooDOwjCNK7tRSU8KulBEGMEMrvnScbNmoDvwYwUZFVK8tW8MSeWH+pbD8GR6rJjSmiTikkH1ChZI6o+A+AkHhVaNoH044Ddea6x3CvwEYO9HXOADmZZfwUgx2cym6V6D/jw4QtqriXCdspf root@trusty-patched
-----END SSH HOST KEY KEYS-----
Cloud-init v. 0.7.5 finished at Tue, 22 Sep 2015 02:41:54 +0000. Datasource DataSourceOpenStack [net,ver=2]. Up 30.81 seconds

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Stéphane Graber(stgraber) wrote on 2015-09-22T16:37:21.923910+00:00

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2015-09-22T16:47:19.905185+00:00

This bug was fixed in the package cloud-init - 0.7.5-0ubuntu1.11

cloud-init (0.7.5-0ubuntu1.11) trusty; urgency=medium

[ Felipe Reyes ]

  • d/patches/fix-consumption-of-vendor-data.patch:
    • Fix consumption of vendor-data in OpenStack to allow namespacing
      (LP: #1469260).

[ Scott Moser ]

  • d/patches/lp-1461242-generate-ed25519-host-keys.patch:
    • ssh: generate ed25519 host keys if supported (LP: #1461242)

-- Scott Moser smoser@ubuntu.com Fri, 11 Sep 2015 20:22:00 -0400

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
launchpad Migrated from Launchpad
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ubuntu-server-builder