Regression between 0.12.24 and 0.12.29 on ec2metadata calls

[lifeofguenter]

It seems like 0.12.29 is doing a lot more requests to ec2metadata than 0.12.24 to a point where we are being rate limited (see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html)

It is advised:

• cache results
• implement retry with back-off

Logs:

[2020-08-13T08:20:39.133Z] 2020/08/13 08:20:38 [INFO] Setting AWS metadata API timeout to 100ms
[2020-08-13T08:20:39.702Z] 2020/08/13 08:20:39 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
[2020-08-13T08:20:39.712Z] 2020/08/13 08:20:39 [INFO] Attempting to use session-derived credentials
[2020-08-13T08:20:40.293Z] 2020/08/13 08:20:40 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
[2020-08-13T08:20:40.304Z] 2020/08/13 08:20:40 [INFO] Attempting to use session-derived credentials

[ghost]

This issue has been automatically migrated to hashicorp/terraform-provider-aws#14624 because it looks like an issue with that provider. If you believe this is not an issue with the provider, please reply to hashicorp/terraform-provider-aws#14624.

[lifeofguenter]

@jbardin please re-open, this has nothing to do with the aws terraform provider :)

By using the following proxy: https://github.com/lifeofguenter/ec2metadata-proxycache, I was able to peek into the requests and have the following findings:

 cat tf-0-12-24.log  | cut -d" " -f 7 | sort | uniq -c | sort -bgr
11368 /latest/meta-data/instance-id
11316 /latest/meta-data/iam/security-credentials/
11294 /latest/meta-data/iam/security-credentials/backoffice_tidalmusic_jenkins_ops_node
 102 /latest/api/token

vs.

 cat tf-0-12-29.log  | cut -d" " -f 7 | sort | uniq -c | sort -bgr
11503 /latest/api/token
11309 /latest/meta-data/instance-id
11301 /latest/meta-data/iam/security-credentials/
11286 /latest/meta-data/iam/security-credentials/backoffice_tidalmusic_jenkins_ops_node

Notice how with 0.12.29 (no code changes, no provider changes) there are ~10k more requests towards the instance-metadata service. It seems most of the gain is going towards /api/token.

These number of requests are immense and can most probably be cached as they do not change within a single terraform process run.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.