Support IMDSv2 for EC2 Discovery

[ashwin316]

We have been using Elasticsearch for storing our application data, and are extensively using the EC2 discovery plugin and S3 plugin. As per AWS best practices, we are making use of the EC2 instance metadata to get the required auth tokens for EC2 discovery and S3 snapshots, and dont make use of any access_keys and secret_access_keys. Following is the elasticsearch.yml file:

discovery.seed_providers: ec2
discovery.ec2.tag.cluster_name: elasticsearch
network.host: _ec2:privateIpv4_
http.port: 9200
discovery.ec2.endpoint: ec2.ap-south-1.amazonaws.com
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
logger.org.elasticsearch.discovery.ec2: "WARN"

Recently during our security audit, we got to know about a new Session oriented way of interacting with the EC2 metadata plugin. More info here. As this was a more secured way, we tried it out in our lower environments, which led to our Elasticsearch clusters go into YELLOW state. The reason for this is that the nodes were not able to get the availability zone by querying the instance metadata service.

Logs for the same:

{"type": "server", "timestamp": "2021-11-05T07:31:59,234Z", "level": "DEBUG", "component": "o.e.d.e.Ec2DiscoveryPlugin", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "obtaining ec2 [placement/availability-zone] from ec2 meta-data url http://169.254.169.254/latest/meta-data/placement/availability-zone" } {"type": "server", "timestamp": "2021-11-05T07:31:59,321Z", "level": "ERROR", "component": "o.e.d.e.Ec2DiscoveryPlugin", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-2", "message": "failed to get metadata for [placement/availability-zone]", "stacktrace": ["java.io.IOException: Server returned HTTP response code: 401 for URL: http://169.254.169.254/latest/meta-data/placement/availability-zone",

I am not really sure if this is already implemented or not, as I could not find any documentation for the same. It would be great if we could support this new way of authorization, if not supported yet :)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[ashwin316]

I tried to dig in a bit further, and found that we can fetch the token and use the same token to query the metadata service. As per AWS Documentation, we can use the below requests for successful request using IMDSv2

Fetch Token: TOKEN=curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"
Metadata request: curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/

When I checked the code for the EC2 plugin, I think we can get and append the Token in the request made here

[stretyak28]

Could you answer when there will be support for IMDSv2, since I also encountered the situation that was described above

[basil-lion]

Also have this issue. Are there any timelines fixing it, please?

[arteam]

Hi everyone! I'm working on this, am going to send a pull request with a fix this week.

[SirSmartyPants]

Will this fix be applied to the repository-s3 plugin as well? We've found that snapshots to S3 no longer function after switching to IMDSv2, unless access and secret keys are added to the keystore.

[arteam]

Will this fix be applied to the repository-s3 plugin as well? We've found that snapshots to S3 no longer function after switching to IMDSv2, unless access and secret keys are added to the keystore.

I believe repository-s3 uses the AWS SDK under the hood for all operation, so it should support IMDSv2 out of the box. @SirSmartyPants Would you mind creating a separate issue for that?