3.6.4 does not fetch iam credentials using IMDSv2 when running from inside containers with IMDSv2 Defaults

[v-kumar]

xxxxe.net failed | msg: non-zero return code | stderr: [fog][WARNING] Unable to fetch credentials: read timeout reached
rake aborted!
ArgumentError: Missing required arguments: aws_access_key_id, aws_secret_access_key
/usr/local/bundle/ruby/2.6.0/gems/fog-core-2.2.0/lib/fog/core/service.rb:244:in `validate_options'
/usr/local/bundle/ruby/2.6.0/gems/fog-core-2.2.0/lib/fog/core/service.rb:268:in `handle_settings'
/usr/local/bundle/ruby/2.6.0/gems/fog-core-2.2.0/lib/fog/core/service.rb:98:in `new'
/usr/local/bundle/ruby/2.6.0/gems/fog-core-2.2.0/lib/fog/core/services_mixin.rb:29:in `new'
/usr/local/bundle/ruby/2.6.0/gems/asset_sync-2.11.0/lib/asset_sync/storage.rb:20:in `connection'

3.6.3 works fine with instance credentials, however 3.6.4 does not.

[v-kumar]

I just found the reason. Our instances still use IMDSv1. They have not yet been configured with IMDSV2. So, anybody who uses the latest GEM without configuring for IMDSv2 would fail. This is a high severity bug, IMO.

Gem should check if the instance is configured for IMDSv1 or IMDSv2 and use the appropriate one as IMDSv2 disables IMDSv1.

[geemus]

Thanks for the heads up, I had been under the impression that this change was backwards compatible and unfortunately that does not sound like it is the case.

Is there an easy way to check for versions to configure this on the server side? I don't have recent direct experience with this, so I've been going off the docs and what others have reported.

@atyndall helped add v2 stuff. @atyndall Do you have thoughts on how best to fix this?

[atyndall]

This is curious, my understanding of the docs (and my own testing) has indicated that AWS has simply turned on IMDSv2 for all existing instances. As there are no instructions on how to "enable IMDSv2", just how to disable IMDSv1, and it says "By default, you can use either IMDSv1 or IMDSv2, or both." - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html

When I switched my instances over to IMDSv2, I did not have to enable anything. The only proactive step was when I was satisfied I had switched over, was to disable IMDSv1.

@v-kumar can you provide more details on your setup? Is it possible you are mocking the IMDS service and not directly communicating with the official AWS version? Are you using a really long-lived EC2 instance or something like that?

@geemus As for how to fix, it should be as simple as not including the token in the request if the request to retrieve it fails. I can prepare a PR on Monday to get this resolved if you would like?

[geemus]

@atyndall thanks for the quick response. My reading of things matches your suggestion, so I was a bit surprised as well. If we haven't heard more and this still seems to be an issue, doing a fix along those lines early next week sounds reasonable to me. Thanks!

[v-kumar]

My instance were not switched to IMDSv2 automatically. Their documentation does not indicate that as well.

Per their documentation existing instances must be configured to use IMDSv2. See **Step 3 ** under Transitioning to Version 2

They say

For existing instances: You can require IMDSv2 use through the modify-instance-metadata-options command. You can make these changes on running instances; you do not need to restart your instances.

Also. please note that AWS sometimes does these defaults based on Unix flavor - IMDSv2 could be default for AWS Linux/Linux 2, but not for Ubuntu.

[atyndall]

@v-kumar Your quoted paragraph refers to the fact that you can configure AWS to require IMDSv2 (e.g. disable IMDSv1), which is what I have tested and verified that command does. The docs are quite clear that by default both versions are always available to be used.

My testing was also performed on Ubuntu 18.04 and 20.04 as that's the system base I run.

There is also no indication in the docs that the metadata API endpoint would change based on AMI choice, which makes sense as my understanding is that it's a hypervisor feature.

Are you running an EBS or Instance Store based AMI?

Can you provide more information on your specific setup?

Can you provide the full output of running curl -v -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 1" and curl -v http://169.254.169.254/latest/meta-data/ on your affected instance?

Thanks

[v-kumar]

Actually, my bad, though not so simple.

This curl -v http://169.254.169.254/latest/meta-data/ works in both my instance as well as docker containers within the instance.

This works only within the instance curl -v -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 1", but does not work from within any docker container node running on the instance. It just hangs. I have tried this in few different containers and same effect.

[v-kumar]

Bit more. I found that botocore users ran into the same issue.

I do believe that, just like botocore did, the right thing to do is revert the change as the side effects are esoteric and hard to debug.

[v-kumar]

I am renaming the title of the issue...

[v-kumar]

I was able to verify that the latest release works from within containers after I set the hop limit.

aws ec2 modify-instance-metadata-options --instance-id "{{item}}" --http-put-response-hop-limit 2 --http-endpoint enabled

[atyndall]

Thanks for the further info, I’ll prepare a PR on Monday (Australian Time) to fallback to V1 using a similar algorithm to aws-sdk-core, which involves some retry logic on V2 error.

Moving forward, as you’ve discovered, setting the hop limit higher than the default 1 in the case you are using virtualised networking within your instance (Docker, etc) will allow you to take advantage of the V2 API and its security benefits moving forward.

[v-kumar]

Thank you @atyndall

[geemus]

Thanks to you both for discussing and working through this. Glad to hear we were able to pin it down and have a plan for fixing it next week.

[atyndall]

@geemus @v-kumar I have created PR #562 to address this