EC2 credentials are cached on disk

[ubuntu-server-builder]

This bug was originally filed in Launchpad as LP: #1638312

Launchpad details

affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)', 'cloud-init (Ubuntu Yakkety)']
assignee = None
assignee_name = None
date_closed = 2017-09-23T02:14:14.426176+00:00
date_created = 2016-11-01T14:49:42.830432+00:00
date_fix_committed = 2017-01-20T18:50:35.714003+00:00
date_fix_released = 2017-09-23T02:14:14.426176+00:00
id = 1638312
importance = medium
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1638312
milestone = None
owner = smoser
owner_name = Scott Moser
private = False
status = fix_released
submitter = smoser
submitter_name = Scott Moser
tags = ['verification-done-xenial', 'verification-done-yakkety']
duplicates = []

Launchpad user Scott Moser(smoser) wrote on 2016-11-01T14:49:42.830432+00:00

=== Begin SRU Template ===
[Impact]
On EC2, instance metadata can include credentials that remain valid for as
much as 6 hours. Reading these and allowing them to be pickled represents a
potential vulnerability if a snapshot of the disk is taken and shared as part
of an AMI.

The fix applied was simply to avoid reading the security credentials
in cloud-init.

[Test Case]

1. Launch an instance on Ec2.
2. Verify broken-ness by verifying 'security-credentials' exists in the
      pickled object in /var/lib/cloud/instance/obj.pkl
3. enable proposed, update, upgrade
4. clean instance
      rm -Rf /var/lib/cloud /var/log/cloud-init*
5. reboot
6. go back in and verify no 'security-credentials' are present.

[Regression Potential]
Low, but possible if someone was using the obj.pkl and expecting to
find credentials there. No one should be doing that.

Second possibility is if someone was using cloud-init's
get_instance_metadata and expected to have the security-credentials there.

=== End SRU Template ===
On EC2, instance metadata can include credentials that remain valid for as much
as 6 hours. Reading these and allowing them to be pickled represents a
potential vulnerability if a snapshot of the disk is taken and shared as part
of an AMI.

Note that:
a.) the simple fact of storing the credentials in a file that is readable only by root is not a serious problem as any attacker on the system has access to the network available data.
b.) General care needs to be taken for anyone "capturing" an ami and then making it public.

the suggested fix is to skip security-credentials when walking the meta-data tree.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-11-01T14:56:04.793516+00:00

Attaching the suggested fix per Andrew. I fixed flake8 complaints in the test, but that is all.

Launchpad attachments: suggested fix [per andrew]

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-11-01T19:07:02.098764+00:00

Andrew,
Please let me know if you want some path for this other than me making this a public bug and fixing.

[ubuntu-server-builder]

Launchpad user Andrew Jorgensen(ajorgens) wrote on 2016-11-01T19:32:01.535876+00:00

Hi Scott,
Anthony Liguori will coordinate this with you.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2016-11-04T15:56:45.779861+00:00

Anthony, Andrew.
I'd fix this now, and we could start getting it back into 16.04 and the like, I'm waiting on you though.

[ubuntu-server-builder]

Launchpad user Andrew Jorgensen(ajorgens) wrote on 2017-01-13T20:32:02.933649+00:00

I'm not sure why (or if?) folks on Amazon's side have dropped the ball here, but please go ahead and publish this fix at your convenience, if you haven't yet.

[ubuntu-server-builder]

Launchpad user Seth Arnold(seth-arnold) wrote on 2017-01-13T21:18:56.006026+00:00

Has a CVE been assigned to this issue? Or does it fall into the category of "security hardening" and thus not qualify for a CVE?

Thanks

[ubuntu-server-builder]

Launchpad user Andrew Jorgensen(ajorgens) wrote on 2017-01-13T21:33:11.870221+00:00

No CVE has been assigned, and in fact it seems only Amazon Linux was vulnerable, because Ubuntu and others were using an API version for instance metadata that did not include IAM instance credentials.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2017-02-04T03:49:52.859201+00:00

This bug was fixed in the package cloud-init - 0.7.9-19-ge987092-0ubuntu1

---

cloud-init (0.7.9-19-ge987092-0ubuntu1) zesty; urgency=medium

• New upstream snapshot.
  • manual_cache_clean: When manually cleaning touch a file in instance dir.
  • Add tools/ds-identify to identify datasources available.
  • Fix small typo and change iso-filename for consistency [Robin Naundorf]
  • Fix eni rendering of multiple IPs per interface
    [Ryan Harper] (LP: #1657940)
  • tools/mock-meta: support python2 or python3 and ipv6 in both.
  • tests: remove executable bit on test_net, so it runs, and fix it.
  • tests: No longer monkey patch httpretty for python 3.4.2
  • Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
    [Lars Kellogg-Stedman] (LP: #1658174)
  • reset httppretty for each test [Lars Kellogg-Stedman] (LP: #1658200)
  • build: fix running Make on a branch with tags other than master
  • EC2: Do not cache security credentials on disk
    [Andrew Jorgensen] (LP: #1638312)
  • doc: Fix typos and clarify some aspects of the part-handler
    [Erik M. Bray]
  • doc: add some documentation on OpenStack datasource.
  • OpenStack: Use timeout and retries from config in get_data.
    [Lars Kellogg-Stedman] (LP: #1657130)
  • Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
  • Fix minor docs typo: perserve > preserve [Jeremy Bicha]
  • Use dnf instead of yum when available [Lars Kellogg-Stedman]
    (LP: #1647118)
  • validate-yaml: use python rather than explicitly python3
  • Get early logging logged, including failures of cmdline url.

-- Scott Moser smoser@ubuntu.com Fri, 03 Feb 2017 21:54:39 -0500

[ubuntu-server-builder]

Launchpad user Jon Grimm(jgrimm) wrote on 2017-03-03T20:26:55.622898+00:00

Are we able to turn this bug public?

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2017-03-07T23:24:41.030694+00:00

Hello Scott, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-48-g1c795b9-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Chris Halse Rogers(raof) wrote on 2017-03-07T23:33:23.061196+00:00

Hello Scott, or anyone else affected,

Accepted cloud-init into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.9-48-g1c795b9-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2017-03-08T19:37:13.853962+00:00

In order to see the security credentials, you must launch an instance with
iam role attached (--iam-instance-profile). Other wise they do not show up in the metadata service.

Ubuntu cloud-init did not actually show this bug because it read and stored
a version of the metadata service (2009-04-04) that does not have the iam/
credentials.

This can be verified simply by
$ sudo python3 -c 'from cloudinit.stages import _pkl_load as pl; import pprint; pprint.pprint(pl("/var/lib/cloud/instance/obj.pkl").metadata)'
{'ami-id': 'ami-78b7166e',
'ami-launch-index': '0',
'ami-manifest-path': '(unknown)',
'block-device-mapping': {'ami': '/dev/sda1', 'root': '/dev/sda1'},
'hostname': 'ip-10-0-0-160',
'instance-action': 'none',
'instance-id': 'i-00e90430c5dbe321c',
'instance-type': 't2.micro',
'local-hostname': 'ip-10-0-0-160',
'local-ipv4': '10.0.0.160',
'placement': {'availability-zone': 'us-east-1b'},
'profile': 'default-hvm',
'public-hostname': b'',
'public-ipv4': '54.86.100.172',
'public-keys': {'brickies': ['ssh-rsa '
'AAAAB3NzaC1yc2EAAAABIwAAAQEA3I7VUf2l5gSn5uavROsc5HRDpZdQueUq5ozemNSj8T7enqKHOEaFoU2VoPgGEWC9RyzSQVeyD6s7APMcE82EtmW4skVEgEGSbDc1pvxzxtchBj78hJP6Cf5TCMFSXw+Fz5rF1dR23QDbN1mkHs7adr8GW4kSWqU7Q7NDwfIrJJtO7Hi42GyXtvEONHbiRPOe8stqUly7MvUoN+5kfjBM8Qqpfl2+FNhTYWpMfYdPUnE7u536WqzFmsaqJctz3gBxH9Ex7dFtrxR4qiqEr9Qtlu3xGn7Bw07/+i1D+ey3ONkZLN+LQ714cgj8fRS4Hj29SCmXp5Kt5/82cD/VN3NtHw== '
'brickies']},
'reservation-id': 'r-07f18d71dc1ebd0ab',
'security-groups': 'wide-open'}

However, the get_instance_metadata() function would show the credentials.

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'Code': 'Success', 'LastUpdated': '2017-03-08T19:14:59Z'}, 'security-credentials': {'smtest-ec2-ro': {'Token': 'FQoDY...Bxgu=', 'AccessKeyId': 'XXXXXXXXXXXXXXXXXXXX', 'Expiration': '2017-03-09T01:43:31Z', 'Code': 'Success', 'SecretAccessKey': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'Type': 'AWS-HMAC', 'LastUpdated': '2017-03-08T19:14:40Z'}}}

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2017-03-08T19:38:08.997514+00:00

ami-78b7166e ubuntu/images-testing/hvm-ssd/ubuntu-yakkety-daily-amd64-server-20170307

$ ec2metadata --ami-id
ami-78b7166e

After upgrade..

$ dpkg-query --show cloud-init
cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.10.1

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'Code': 'Success', 'LastUpdated': '2017-03-08T19:33:11Z'}}

Notice that the 'security-credentials' dictionary is not present.

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2017-03-08T19:40:12.527614+00:00

ami-f4cc1de2 ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20170221

$ ec2metadata --ami-id
ami-f4cc1de2

$ dpkg-query --show cloud-init
cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.04.1

$ python3 -c 'from cloudinit import ec2_utils; print(ec2_utils.get_instance_metadata("latest")["iam"])'
{'info': {'Code': 'Success', 'InstanceProfileId': 'AIPAJ7VG3LHE4JOYDGNG6', 'InstanceProfileArn': 'arn:aws:iam::950047163771:instance-profile/smtest-ec2-ro', 'LastUpdated': '2017-03-08T19:15:50Z'}}

Notice that the 'security-credentials' entry is missing from the 'info' dict.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2017-03-16T16:18:49.500475+00:00

This bug was fixed in the package cloud-init - 0.7.9-48-g1c795b9-0ubuntu1~16.04.1

---

cloud-init (0.7.9-48-g1c795b9-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

• debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
• debian/patches/ds-identify-behavior-xenial.patch: adjust default
  behavior of ds-identify for SRU (LP: #1669675, #1660385).
• New upstream snapshot.
  • Support warning if the used datasource is not in ds-identify's list
    (LP: #1669675).
  • DatasourceEc2: add warning message when not on AWS. (LP: #1660385)
  • Z99-cloudinit-warnings: Add profile.d script for showing warnings on
  • Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
    execute bit in permissions.
  • (RedHat) net: correct errors in cloudinit/net/sysconfig.py
    [Lars Kellogg-Stedman]
  • ec2_utils: fix MetadataLeafDecoder that returned bytes on empty
  • Fix eni rendering of multiple IPs per interface [Ryan Harper]
    (LP: #1657940)
  • Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
    [Lars Kellogg-Stedman]
  • EC2: Do not cache security credentials on disk [Andrew Jorgensen]
    (LP: #1638312)
  • OpenStack: Use timeout and retries from config in get_data.
    [Lars Kellogg-Stedman] (LP: #1657130)
  • Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
  • (RedHat) Use dnf instead of yum when available [Lars Kellogg-Stedman]
  • Get early logging logged, including failures of cmdline url.
  • test / doc / build environment changes
    • Remove style checking during build and add latest style checks to
      tox [Joshua Powers]
    • code-style: make master pass pycodestyle (2.3.1) cleanly, currently
      [Joshua Powers]
    • Fix small typo and change iso-filename for consistency
    • tools/mock-meta: support python2 or python3 and ipv6 in both.
    • tests: remove executable bit on test_net, so it runs, and fix it.
    • tests: No longer monkey patch httpretty for python 3.4.2
    • reset httppretty for each test [Lars Kellogg-Stedman]
    • build: fix running Make on a branch with tags other than master
    • doc: Fix typos and clarify some aspects of the part-handler
      [Erik M. Bray]
    • doc: add some documentation on OpenStack datasource.
    • Fix minor docs typo: perserve > preserve [Jeremy Bicha]
    • validate-yaml: use python rather than explicitly python3

-- Scott Moser smoser@ubuntu.com Mon, 06 Mar 2017 16:34:10 -0500

[ubuntu-server-builder]

Launchpad user Brian Murray(brian-murray) wrote on 2017-03-16T16:19:04.964313+00:00

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[ubuntu-server-builder]

Launchpad user Launchpad Janitor(janitor) wrote on 2017-03-16T16:19:33.912058+00:00

This bug was fixed in the package cloud-init - 0.7.9-48-g1c795b9-0ubuntu1~16.10.1

---

cloud-init (0.7.9-48-g1c795b9-0ubuntu1~16.10.1) yakkety; urgency=medium

• debian/rules: install Z99-cloudinit-warnings.sh to /etc/profile.d
• debian/patches/ds-identify-behavior-yakkety.patch: adjust default
  behavior of ds-identify for SRU (LP: #1669675, #1660385).
• New upstream snapshot.
  • Support warning if the used datasource is not in ds-identify's list
    (LP: #1669675).
  • DatasourceEc2: add warning message when not on AWS. (LP: #1660385)
  • Z99-cloudinit-warnings: Add profile.d script for showing warnings on
  • Z99-cloud-locale-test.sh: convert tabs to spaces, remove unneccesary
    execute bit in permissions.
  • (RedHat) net: correct errors in cloudinit/net/sysconfig.py
    [Lars Kellogg-Stedman]
  • ec2_utils: fix MetadataLeafDecoder that returned bytes on empty
  • Fix eni rendering of multiple IPs per interface [Ryan Harper]
    (LP: #1657940)
  • Add 3 ecdsa-sha2-nistp* ssh key types now that they are standardized
    [Lars Kellogg-Stedman]
  • EC2: Do not cache security credentials on disk [Andrew Jorgensen]
    (LP: #1638312)
  • OpenStack: Use timeout and retries from config in get_data.
    [Lars Kellogg-Stedman] (LP: #1657130)
  • Fixed Misc issues related to VMware customization. [Sankar Tanguturi]
  • (RedHat) Use dnf instead of yum when available [Lars Kellogg-Stedman]
  • Get early logging logged, including failures of cmdline url.
  • test / doc / build environment changes
    • Remove style checking during build and add latest style checks to
      tox [Joshua Powers]
    • code-style: make master pass pycodestyle (2.3.1) cleanly, currently
      [Joshua Powers]
    • Fix small typo and change iso-filename for consistency
    • tools/mock-meta: support python2 or python3 and ipv6 in both.
    • tests: remove executable bit on test_net, so it runs, and fix it.
    • tests: No longer monkey patch httpretty for python 3.4.2
    • reset httppretty for each test [Lars Kellogg-Stedman]
    • build: fix running Make on a branch with tags other than master
    • doc: Fix typos and clarify some aspects of the part-handler
      [Erik M. Bray]
    • doc: add some documentation on OpenStack datasource.
    • Fix minor docs typo: perserve > preserve [Jeremy Bicha]
    • validate-yaml: use python rather than explicitly python3

-- Scott Moser smoser@ubuntu.com Mon, 06 Mar 2017 16:37:28 -0500

[ubuntu-server-builder]

Launchpad user Scott Moser(smoser) wrote on 2017-09-23T02:14:17.341715+00:00

This bug is believed to be fixed in cloud-init in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.