Support for Instance Metadata Service Version 2 (IMDSv2) in S3 driver

[Crghilardi]

Currently, the S3 driver only supports accessing role credentials through querying the instance metadata using V1 method

The v2 of IMDS adds a token component that gets added into the headers of subsequent requests.
Many organizations are moving towards v2 after various security incidents.

If you try to run a pipeline on an instance with imdbsv2 enforced , the response (CURL_VERBOSE=1) looks roughly like

> GET /latest/meta-data/iam/security-credentials/HTTP/1.1
Host: 169.254.169.254
Accept:*/*

* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 401 Unauthorized
< Content-Length: 123
< Content-Type: text/html
< Date: ...
< Connection: close
< Server: EC2ws

In the meantime, I believe you should be able to generate the token and manually embed it in a config.json, something like:

{
	"arbiter":
	{"s3": {
		"headers" :
		{"X-aws-ec2-metadata-token" : "ABCDEFGHI"}
		}
	}
}

I don't yet have a fully working example of the above logic, but will continue to tinker and update later.

[Crghilardi]

After looking at this a little more, I don't think this would be immediately supported with a user provided token like I outlined above.

1. The actual call for gathering instance credentials is an Http:: call, so the above config would be using http or curl. Not entirely sure which syntax is used

2. the actual call to the AWS IP doesn't seem to populate any headers, user provided or otherwise. Not sure if this is as easy as just expanding that call out a bit or more logic would have to be added

[Crghilardi]

As a work around, one can still retrieve credentials manually outside of arbiter and pass them in via config file.

TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

CREDS=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/your-role-name-here)

you can then use jq to parse them into a config file

echo $CREDS | jq '.["SecretAccessKey"] as $secret |
				  .["AccessKeyId"] as $access | 
				  .["Token"] as $token |
				  { arbiter:
				  	{s3: {
				  		access: $access,
				  		secret: $secret,
				  		token: $token
				  		}
				  	}
				  }' > config.json