Add support for EC2 IMDSv2 (Instance Metadata Service)

[roadmapper]

Porting the PR from the main Ansible repo: ansible/ansible#68098

SUMMARY

In November 2019, AWS announced a new version of the instance metadata service (IMDSv2) that supports session authentication. Currently, if the ec2_metadata_facts module is used on an EC2 instance that only has IMDSv2 enabled the module will fail. This change adds support for getting the session token for IMDSv2.

Fixes ansible/ansible#67981

When the Instance Metadata Service HTTP endpoint is disabled, the module will fail to get facts:

$ aws ec2 modify-instance-metadata-options --instance-id <instance_id> --http-endpoint disabled
$ ansible all -i 'localhost,' -m ec2_metadata_facts -c local
localhost | FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "msg": "Failed to retrieve metadata token from AWS: HTTP Error 403: Forbidden",
    "response": {
        "body": "",
        "connection": "close",
        "content-length": "0",
        "content-type": "text/plain",
        "date": "Sun, 22 Mar 2020 19:58:36 GMT",
        "msg": "HTTP Error 403: Forbidden",
        "server": "EC2ws",
        "status": 403,
        "url": "http://169.254.169.254/latest/api/token"
    }
}

ISSUE TYPE

• Feature Pull Request

COMPONENT NAME

ec2_metadata_facts

[roadmapper]

@jillr, do you know who I could reach out to to get this PR reviewed? Would love to be able to help get this change as it has been 8 months since Amazon introduced the newer version of the instance metadata service.

[ansibullbot]

@roadmapper this PR contains the following merge commits:

• 051bd73

Please rebase your branch to remove these commits.

click here for bot help

[Lauren-Buchholz-Moxe]

We are having the same issue, would love to get this fix merged and ready

[jgournet]

Is there any way to make this PR progress please ?
CC @jillr

[tremble]

The biggest thing to help get the PR merged would be to add some integration or unit tests. While there's some of the initial framework for an ec2_metadata_facts integration test, I don't see any real tests built out. Because of the way the CI infrastructure is set up unit tests may be simpler to build.

However, I'm also concerned that this looks like a backwards incompatible change. While many of the modules are initially designed and tested against AWS, there are a significant number of IaaS services which support things like the metadata interface but aren't AWS so might not support the IMDSv2 interfaces. (For example OpenStack: https://docs.openstack.org/nova/latest/admin/metadata-service.html) While these aren't the primary use case for this collection there are people who use the modules this way and it's generally preferable to avoid breaking things which did work. Personally I'd prefer if this behaviour was either opt-in, or defaulted to falling back to the v1 service.

[tremble]

@jillr is working on integration tests with #212

[roadmapper]

However, I'm also concerned that this looks like a backwards incompatible change. While many of the modules are initially designed and tested against AWS, there are a significant number of IaaS services which support things like the metadata interface but aren't AWS so might not support the IMDSv2 interfaces. (For example OpenStack: https://docs.openstack.org/nova/latest/admin/metadata-service.html) While these aren't the primary use case for this collection there are people who use the modules this way and it's generally preferable to avoid breaking things which did work. Personally I'd prefer if this behaviour was either opt-in, or defaulted to falling back to the v1 service.

It's been a little while since I've tested this change, but from what I recall: if IMDSv2 is not enabled, requesting the token doesn't do anything (accepted by the metadata service and nothing is returned).

However, that being said, I'd be happy to change the module to accept a flag to opt-in to requesting a token for the IMDS. The point about having AWS-like environments using this module was not something I was aware of and is a good reason to make the feature opt-in.

If the feature is opt-in, should I change to documentation to point out that the use of this module without a flag will mean that if it is used on a IMDSv2 enabled EC2 instance, the module will fail?

[tremble]

I've updated the code to use (200, 404) for this case I was worried about, and added a changelog. If the tests pass I think we're good to merge this PR.

[ansibullbot]

cc @s-hertel @silviud @wimnat
click here for bot help

[tremble]

I've also tested this PR against OpenStack and it fails over cleanly.

[tremble]

Tests have all passed. I've also manually tested this against OpenStack (which doesn't support IMDSv2) and it failed over cleanly.

Thanks for your submission, and I apologise for how long it took to get this PR merged.

[roadmapper]

Thanks for helping to get this merged in! Appreciate the edits @tremble 👏

[ado120]

What version of ansible has these changes?

[tremble]

@ado120 This feature should be available in the 1.5.0 release of this collection. I believe this is available with the Ansible 3.4.0 release: https://groups.google.com/g/ansible-devel/c/1buCqmI253g?pli=1