EC2 metadata retrieval fails with spaces in a resource name

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #962046

Launchpad details

affected_projects = ['python-boto (Ubuntu)', 'python-boto (Ubuntu Precise)', 'python-boto (Ubuntu Quantal)', 'python-boto (Ubuntu Raring)']
assignee = None
assignee_name = None
date_closed = 2012-07-10T21:56:30.701726+00:00
date_created = 2012-03-22T10:37:15.687021+00:00
date_fix_committed = None
date_fix_released = None
id = 962046
importance = low
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/962046
milestone = None
owner = juergh
owner_name = Juerg Haefliger
private = False
status = invalid
submitter = juergh
submitter_name = Juerg Haefliger
tags = ['removal-candidate', 'verification-done']
duplicates = []

Launchpad user Juerg Haefliger(juergh) wrote on 2012-03-22T10:37:15.687021+00:00

[Impact]

Example when using a key name 'my key':

Caught exception reading instance data: http://169.254.169.254/2009-04-04/meta-data/mpi/my key

Is a boto issue, see boto/boto#659

Patches provided for Precise, Quantal and Raring. Fix already exists in Saucy.

[Test Case]

There are two ways to test this. The first provides that the change will affect
the url in the required manner but does not test the package. The second will
test the package but needs to be run within an Openstack instance against an
API that has calls with spaces on their name.

====
#!/usr/bin/env python
import urllib

resource = "http://169.254.169.254/2009-04-04/meta-data/i love spaces"
print "unsafe: %s" % resource

resource = urllib.quote(resource, safe="/:")
print "safe: %s" % resource

===
#!/usr/bin/env python
from boto import utils
utils.get_instance_metadata()

[Regression Potential]

This is a very low risk change which makes any non-alphnumeric characters in a meta api call name safe for urlib. It is unlikely that an api call name would have anything other than alphanumeric chars, whitespace and perhaps '_' but even if they did they would be safe here so there is no known regression potential.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-05-30T14:33:02.316713+00:00

Could you give an example of how you start an instance such that '/meta-data/mpi/my key' exists?
I just tried this with a keypair (euca-import-keypair --public-key-file my.pub "my key", and then run instance with '--key "my key"'), and that worked fine (thi sis actually because of the way that public-keys are displayed and handled specially in the MD).

Anyway, can you explain how you did this so I can reproduce?

[ubuntu-server-builder]

Launchpad user Juerg Haefliger(juergh) wrote on 2012-06-01T07:59:11.459432+00:00

nova boot --flavor <FLAVOR_ID> --image <IMAGE_ID> --key_name 'my key' <INSTANCE_NAME>

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-06-01T12:01:03.456493+00:00

Ah.
I didn't realize that this was when the metadata service was coming from nova.
So, then arguably this is a bug in the metadata service on nova, as I could not reproduce against the MD on EC2.

[ubuntu-server-builder]

Launchpad user Juerg Haefliger(juergh) wrote on 2012-06-04T06:21:30.459184+00:00

What does the MD look like on EC2? Curious, can you send a dump? Yes it seems to be a problem with the nova MD service in that it's different from EC2 but I still think boto should be able to handle special characters in MD URLs or is there an EC2 spec that this will never happen with EC2?

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-06-04T13:20:12.199553+00:00

Hm.. Its interesting. digging further, I don't understand how this caused a problem on nova. It seems (either by design or happenstance) that the general behavior of the metadata service handles this issue, and boto is designed to read the metadata service as it is implemented.

I actually verified functional 'boto.utils.get_instance_metadata' on both nova and ec2 with a key named "my brickies" (nova folsom as packaged in Ubuntu). The output of the commands below was identical.

Example:
$ wget http://169.254.169.254/2009-04-04/meta-data/public-keys -O - -q; echo
0=my brickies
$ wget http://169.254.169.254/2009-04-04/meta-data/public-keys/0/ -O - -q; echo
openssh-key
$ wget http://169.254.169.254/2009-04-04/meta-data/public-keys/0/openssh-key -O - -q;
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA3I7VUf2l5gSn5uavROsc5HRDpZdQueUq5ozemNSj8T7enqKHOEaFoU2VoPgGEWC9RyzSQVeyD6s7APMcE82EtmW4skVEgEGSbDc1pvxzxtchBj78hJP6Cf5TCMFSXw+Fz5rF1dR23QDbN1mkHs7adr8GW4kSWqU7Q7NDwfIrJJtO7Hi42GyXtvEONHbiRPOe8stqUly7MvUoN+5kfjBM8Qqpfl2+FNhTYWpMfYdPUnE7u536WqzFmsaqJctz3gBxH9Ex7dFtrxR4qiqEr9Qtlu3xGn7Bw07/+i1D+ey3ONkZLN+LQ714cgj8fRS4Hj29SCmXp5Kt5/82cD/VN3NtHw== my brickies

So, the public-keys (also in nova) is special cased, and you do not ever refer to the key name when crawling. So the space shouldn't actually be used.

I agree the general suggestion, though, that boto should not be doing requests with unencoded urls.

Launchpad attachments: typescript of some other info 'get_instance_metadata'

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-06-04T13:45:46.234104+00:00

I'm also still confused at the initial report, you have "mpi". On the nova I'm looking at:
$ wget http://169.254.169.254/2009-04-04/ -O - -q ; echo
meta-data/
user-data
$ wget http://169.254.169.254/2009-04-04/meta-data/ -O - -q; echo
reservation-id
public-keys/
security-groups
public-ipv4
ami-manifest-path
instance-type
instance-id
local-ipv4
local-hostname
placement/
ami-launch-index
public-hostname
hostname
ami-id
instance-action
block-device-mapping/

$ wget http://169.254.169.254/2009-04-04/meta-data/mpi -O -
--2012-06-04 13:41:23-- http://169.254.169.254/2009-04-04/meta-data/mpi
Connecting to 169.254.169.254:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2012-06-04 13:41:23 ERROR 404: Not Found.

Attemping to import a 'my brickies' keypair name via either web ui or command line client on HP cloud gives:
$ nova keypair-add "my brickies" --pub_key ~/.ssh/id_rsa-smoser@brickies.pub
ERROR: Invalid keypair name (HTTP 400)

[ubuntu-server-builder]

Launchpad user Juerg Haefliger(juergh) wrote on 2012-06-07T12:18:46.794149+00:00

Hmm... On the hp cloud I get:

ubuntu@juergh-8419-5531:~$ curl 169.254.169.254/2009-04-04/meta-data
local-ipv4
reservation-id
local-hostname
placement/
ami-launch-index
public-hostname
mpi/
hostname
ami-id
public-keys/
instance-action
public-ipv4
block-device-mapping/
ami-manifest-path
security-groups
instance-type
instance-id

We 'fixed' the issue by not allowing spaces in key names anymore.

[ubuntu-server-builder]

Launchpad user Juerg Haefliger(juergh) wrote on 2012-06-07T12:31:47.131021+00:00

mpi/ was present in diablo but removed subsequently:

commit 8ecdc44690ced882205112e017f79dc98cd6aaca
Author: Jesse Andrews anotherjesse@gmail.com
Date: Tue Mar 6 20:49:16 2012 -0800

remove undocumented, unused mpi 'extension' to ec2 metadata

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-07-10T21:56:05.453508+00:00

This is fixed in upstream boto at boto/boto@fa10336
So any release of boto after 2012-06-15 should have it.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2012-07-10T21:56:30.143101+00:00

Marking 'invalid' on cloud-init, because it will be appropriately fixed in boto.

[ubuntu-server-builder]

Launchpad user Chris J Arges(arges) wrote on 2013-02-21T17:34:28.945089+00:00

Here are some packages with the backported fix:
http://people.canonical.com/~arges/lp962046/

Please test and verify if this works.

[ubuntu-server-builder]

Launchpad user Juerg Haefliger(juergh) wrote on 2013-05-13T08:17:50.441695+00:00

Sorry this took so long. Finally found the time to test the 12.04 package and it works. Thanks.

[ubuntu-server-builder]

Launchpad user Edward Hope-Morley(hopem) wrote on 2013-06-07T13:03:59.067140+00:00

Launchpad attachments: fix-lp962046-python-boto-precise.debdiff

[ubuntu-server-builder]

Launchpad user Edward Hope-Morley(hopem) wrote on 2013-06-07T13:04:17.644711+00:00

Launchpad attachments: fix-lp962046-python-boto-quantal.debdiff

[ubuntu-server-builder]

Launchpad user Edward Hope-Morley(hopem) wrote on 2013-06-07T13:35:32.713958+00:00

[Impact]

This is a very low risk change. It has been tested on Precise. A quantal patch is also provided.

[Test Case]

#!/usr/bin/env python
from boto import utils
utils.get_instance_metadata()

[Regression Potential]

• not likely.

[ubuntu-server-builder]

Launchpad user Edward Hope-Morley(hopem) wrote on 2013-06-07T13:40:30.803012+00:00

#wosru

[ubuntu-server-builder]

Launchpad user Edward Hope-Morley(hopem) wrote on 2013-06-20T10:44:27.759425+00:00

Launchpad attachments: fix-lp962046-python-boto-raring.debdiff

[ubuntu-server-builder]

Launchpad user Edward Hope-Morley(hopem) wrote on 2013-06-20T11:13:50.453670+00:00

updated SRU testcase and added debdiff for Raring

[ubuntu-server-builder]

Launchpad user Martin Pitt(pitti) wrote on 2013-06-24T12:35:35.855210+00:00

This was uploaded and is sitting in the SRU review queues, unsubscribing sponsors.

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2013-06-27T16:52:06.818022+00:00

Hello Juerg, or anyone else affected,

Accepted python-boto into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-boto/2.3.0-1ubuntu0.13.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2013-06-27T16:58:58.865759+00:00

Hello Juerg, or anyone else affected,

Accepted python-boto into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-boto/2.3.0-1ubuntu0.12.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2013-06-27T17:01:45.428614+00:00

Hello Juerg, or anyone else affected,

Accepted python-boto into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/python-boto/2.2.2-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Ubuntu Foundations Team Bug Bot(crichton) wrote on 2013-09-26T19:29:53.510591+00:00

The fix for this bug has been awaiting testing feedback in the -proposed repository for precise for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2013-10-08T12:25:42.819416+00:00

I've marked this verification done for Juerg, per an email on the subject.
Juerg originally reported, and works on the team at HP where this bug was found.

As the problem was/is well understood and the fix both accepted upstream boto and minimally invasive, hopefully this report can be accepted as 'verification-done'.

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2013-10-09T00:17:39.438996+00:00

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2013-10-09T00:18:14.142521+00:00

This bug was fixed in the package python-boto - 2.3.0-1ubuntu0.13.04.1

---

python-boto (2.3.0-1ubuntu0.13.04.1) raring; urgency=low

• Fix failures when there are spaces in a resource name. (LP: #962046)
  -- Chris J Arges chris.j.arges@canonical.com Fri, 08 Feb 2013 09:50:51 -0600

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2013-10-09T01:10:25.018734+00:00

This bug was fixed in the package python-boto - 2.3.0-1ubuntu0.12.10.1

---

python-boto (2.3.0-1ubuntu0.12.10.1) quantal; urgency=low

• Fix failures when there are spaces in a resource name. (LP: #962046)
  -- Chris J Arges chris.j.arges@canonical.com Fri, 08 Feb 2013 09:50:51 -0600

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2013-10-09T01:11:16.656965+00:00

This bug was fixed in the package python-boto - 2.2.2-0ubuntu3

---

python-boto (2.2.2-0ubuntu3) precise; urgency=low

• Fix failures when there are spaces in a resource name. (LP: #962046)
  -- Chris J Arges chris.j.arges@canonical.com Fri, 08 Feb 2013 09:23:57 -0600