New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EC2 credentials discovery fails (instance-data host gone?) #271

Open
screamish opened this Issue Feb 11, 2016 · 13 comments

Comments

Projects
None yet
7 participants
@screamish

screamish commented Feb 11, 2016

I'm trying to use Discover for credentials from on an AWS EC2 instance and it's throwing this exception:

"MissingFileError \"/root/.aws/credentials\"

However I've had a look at the code that's doing this and this function seems like Amazon might have broken a dependency.

https://github.com/brendanhay/amazonka/blob/develop/amazonka/src/Network/AWS/EC2/Metadata.hs#L282

When I try to run this via ssh on my EC2 instance I get the following:

$ curl http://instance-data/latest/meta-data/
curl: (6) Could not resolve host: instance-data

but if I try the following it works:

$ curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
... snip ...

I couldn't find anything on AWS's docs about them removing that hostname. What do you think?

@brendanhay

This comment has been minimized.

Show comment
Hide comment
@brendanhay

brendanhay Feb 12, 2016

Owner

That is concerning. The instance-data name resolution is used because it's convenient for the EC2 test, if that no longer existed I'd need some alternative way of 'quickly' determining if the host is in EC2 or not.

Which region and instance type are you encountering this issue on?

Owner

brendanhay commented Feb 12, 2016

That is concerning. The instance-data name resolution is used because it's convenient for the EC2 test, if that no longer existed I'd need some alternative way of 'quickly' determining if the host is in EC2 or not.

Which region and instance type are you encountering this issue on?

@screamish

This comment has been minimized.

Show comment
Hide comment
@screamish

screamish Feb 12, 2016

ap-southeast-2, t2.small

Is there a reason you can't just use this:

latest :: Text
latest = "http://169.254.169.254/latest/"

like you do in all the other calls?

screamish commented Feb 12, 2016

ap-southeast-2, t2.small

Is there a reason you can't just use this:

latest :: Text
latest = "http://169.254.169.254/latest/"

like you do in all the other calls?

@brendanhay

This comment has been minimized.

Show comment
Hide comment
@brendanhay

brendanhay Feb 12, 2016

Owner

Because it relies on name resolution failure, instead of a timeout. If 169.254.169.254 is used, a timeout needs to be set for http-client, with circumstances such as an overloaded host resulting in the timeout being reached, and the instance not being detected as an EC2 host. I'm open to other suggestions, but I'd prefer to avoid the timeout if possible.

In the interim, if you're deploying to EC2 I'd suggest using newEnv Sydney (FromProfile "default"), or whatever the default IAM profile on that host happens to be. This way the isEC2 check will be skipped entirely.

Owner

brendanhay commented Feb 12, 2016

Because it relies on name resolution failure, instead of a timeout. If 169.254.169.254 is used, a timeout needs to be set for http-client, with circumstances such as an overloaded host resulting in the timeout being reached, and the instance not being detected as an EC2 host. I'm open to other suggestions, but I'd prefer to avoid the timeout if possible.

In the interim, if you're deploying to EC2 I'd suggest using newEnv Sydney (FromProfile "default"), or whatever the default IAM profile on that host happens to be. This way the isEC2 check will be skipped entirely.

@screamish

This comment has been minimized.

Show comment
Hide comment
@screamish

screamish Feb 13, 2016

Ah, thanks for the explanation, that makes good sense. And thanks for the workaround, it covers my use case since I don't need to deploy this multi-region. If I think of any ideas for quick, robust way to implement isEC2 I'll let you know.

screamish commented Feb 13, 2016

Ah, thanks for the explanation, that makes good sense. And thanks for the workaround, it covers my use case since I don't need to deploy this multi-region. If I think of any ideas for quick, robust way to implement isEC2 I'll let you know.

@gregwebs

This comment has been minimized.

Show comment
Hide comment
@gregwebs

gregwebs Aug 17, 2016

I just ran into this same issue. I would suggest not worrying about the overloaded host issue so much unless you have good reason to. My reasoning is that generally finding credentials is done at startup. If the host is already overloaded and you start a new process you are probably in for trouble already.

gregwebs commented Aug 17, 2016

I just ran into this same issue. I would suggest not worrying about the overloaded host issue so much unless you have good reason to. My reasoning is that generally finding credentials is done at startup. If the host is already overloaded and you start a new process you are probably in for trouble already.

@brendanhay

This comment has been minimized.

Show comment
Hide comment
@brendanhay

brendanhay Aug 17, 2016

Owner

Another issue with timeout is after searching for relevant environment variables or ~/.aws/credentials, the isEC2 check is performed before credential discovery fails.

On a development machine that incorrectly sets environment or credential information, failure will always block for the timeout length.

Owner

brendanhay commented Aug 17, 2016

Another issue with timeout is after searching for relevant environment variables or ~/.aws/credentials, the isEC2 check is performed before credential discovery fails.

On a development machine that incorrectly sets environment or credential information, failure will always block for the timeout length.

@brendanhay

This comment has been minimized.

Show comment
Hide comment
@brendanhay

brendanhay Oct 22, 2016

Owner

I'm going to share some information here from a colleague of mine who discovered what is likely the root cause and one possible solution:

It appears instance-data doesn't resolve inside VPCs that have not been configured using enableDnsSupport and enableDnsHostnames. If you can set both of these parameters for your specific VPC then the isEC2 check will work and instance-data will resolve, otherwise it's still recommended to use the workaround above or to preload the EC2 instance check using newEnvWith Oregon Discover (Just True) ....

Owner

brendanhay commented Oct 22, 2016

I'm going to share some information here from a colleague of mine who discovered what is likely the root cause and one possible solution:

It appears instance-data doesn't resolve inside VPCs that have not been configured using enableDnsSupport and enableDnsHostnames. If you can set both of these parameters for your specific VPC then the isEC2 check will work and instance-data will resolve, otherwise it's still recommended to use the workaround above or to preload the EC2 instance check using newEnvWith Oregon Discover (Just True) ....

@LoicAG

This comment has been minimized.

Show comment
Hide comment
@LoicAG

LoicAG Nov 9, 2016

@brendanhay Thanks for the explanation, I was wondering why instance-data wasn't resolving from within our new VPC!

LoicAG commented Nov 9, 2016

@brendanhay Thanks for the explanation, I was wondering why instance-data wasn't resolving from within our new VPC!

@AlistairB

This comment has been minimized.

Show comment
Hide comment
@AlistairB

AlistairB Aug 29, 2017

I'm having some trouble with this when running on the ec2 instance.

  • Discover - MissingFileError "/home/appuser/.aws/credentials"
  • FromProfile "default" - Essentially this won't work because the role name is autogenerated by our deployment tool
  • newEnvWith Discover (Just True) httpManager - MissingFileError "/home/appuser/.aws/credentials"

Is there some way to do this when the role name is not known?

http://169.254.169.254/latest/meta-data/iam/security-credentials/ does return only one role which is correct.

AlistairB commented Aug 29, 2017

I'm having some trouble with this when running on the ec2 instance.

  • Discover - MissingFileError "/home/appuser/.aws/credentials"
  • FromProfile "default" - Essentially this won't work because the role name is autogenerated by our deployment tool
  • newEnvWith Discover (Just True) httpManager - MissingFileError "/home/appuser/.aws/credentials"

Is there some way to do this when the role name is not known?

http://169.254.169.254/latest/meta-data/iam/security-credentials/ does return only one role which is correct.

@kim

This comment has been minimized.

Show comment
Hide comment
@kim

kim Aug 29, 2017

Contributor

@AlistairB did you check whether instance-data resolves on your instances (see #271 (comment))?

Note that, contrary to what the documentation suggests, newEnvWith ... (Just True) ... will execute isEC2 if the first argument is Discover (via getAuth). This may be considered a bug, but it's not obvious how to fix it properly.

However, you can find out the role at runtime (and before initializing your Env) via metadata.

Contributor

kim commented Aug 29, 2017

@AlistairB did you check whether instance-data resolves on your instances (see #271 (comment))?

Note that, contrary to what the documentation suggests, newEnvWith ... (Just True) ... will execute isEC2 if the first argument is Discover (via getAuth). This may be considered a bug, but it's not obvious how to fix it properly.

However, you can find out the role at runtime (and before initializing your Env) via metadata.

@AlistairB

This comment has been minimized.

Show comment
Hide comment
@AlistairB

AlistairB Aug 30, 2017

@kim Thanks for the comment. Didn't realise it will still run the isEC2 logic.

Yes instance-data does not resolve and I cannot change the VPC settings.

Good point, I will manually hit the metadata endpoint to retrieve the role and create env with FromProfile.

AlistairB commented Aug 30, 2017

@kim Thanks for the comment. Didn't realise it will still run the isEC2 logic.

Yes instance-data does not resolve and I cannot change the VPC settings.

Good point, I will manually hit the metadata endpoint to retrieve the role and create env with FromProfile.

@screamish

This comment has been minimized.

Show comment
Hide comment
@screamish

screamish Aug 30, 2017

You can just create an entry in your /etc/hosts for instance-data to the metadata ip, ie, 169.254.169.254, and it should just work. You don't have to go fiddling with the DNS for your whole VPC. That's how we're working around this issue for now.

screamish commented Aug 30, 2017

You can just create an entry in your /etc/hosts for instance-data to the metadata ip, ie, 169.254.169.254, and it should just work. You don't have to go fiddling with the DNS for your whole VPC. That's how we're working around this issue for now.

@blender

This comment has been minimized.

Show comment
Hide comment
@blender

blender Aug 30, 2017

I am experiencing a similar issue (maybe same) on my machine and others' machines too (MacOS) see blender/Rome#102

Amazonka 1.4.5
seems like newEnv Discover is not working properly when ~/.aws/credentials file is missing and trying to get the default env variables.

Disregard. It's working.

blender commented Aug 30, 2017

I am experiencing a similar issue (maybe same) on my machine and others' machines too (MacOS) see blender/Rome#102

Amazonka 1.4.5
seems like newEnv Discover is not working properly when ~/.aws/credentials file is missing and trying to get the default env variables.

Disregard. It's working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment