Prometheus 2.0 prioritizes EC2 instance role over credentials in environment variables

[moshebs]

I was using Prometheus 1.8 with EC2 discovery, and I gave Prometheus the AWS access key using environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY), as documented here - everything worked fine.

When I moved to Prometheus 2.0, this stopped working and I found out Prometheus ignores the environment variables and tries to use the EC2 instance role. since I my EC2 instance role did not have the required permissions, it failed.

Is this the expected behavior? If it is, I think it should be documented in the migration guide.

[brian-brazil]

AWS is a mish-mash of I believe 5 auth methods currently. For normal developers they know what they want, whereas we have to use heuristics. We also have the general principle that all config should be in the config file, not coming from elsewhere.

Here I'd suggest putting the secrets directly in the config, as I don't see a way for us to correctly choose between two different ways of automatically acquiring credentials from outside the config.

[moshebs]

While I understand why operationally-wise it is simple to have the credentials in the config file, security-wise it is not the best choice and AWS recommend using instance profile. From that perspective I am kind of happy I ran through that because it forces me to use Instance profile and drop the env variables.
You are probably using an AWS SDK, right? so the approach I would choose (maybe this is what you do): take credentials from Prometheus config file, and if it does not exist simply call SDK APIs, and let them take it from there. If this is what you do, maybe AWS SDK changed its behavior?

[brian-brazil]

if it does not exist simply call SDK APIs, and let them take it from there. If this is what you do, maybe AWS SDK changed its behavior?

Two of the auth mechanisms require extra code, when to apply those is the issue here.

[fajpunk]

Does this mean that EC2 target discovery is no longer expected to work by specifying the profile key in the config? For example, here's my config that worked with 1.8, is this no longer expected to work?

    ec2_sd_configs:
      - region: 'us-east-2'
        profile: '<my profile name, defined in ~/.aws/credentials of the user running prometheus>'
        refresh_interval: 60s
        port: 9100

To be clear, I'm not complaining, I'm just looking for clarification on how ec2 target discovery works. I'm glad to submit a PR with doc updates once I understand it :)

[brian-brazil]

How it's meant to work is undefined. The only thing that we can guarantee is that putting credentials in the config file will work, anything beyond that is more complicated.

[jeeyoungk]

I am reverting the PR that caused the change in behavior. See #3985

[moshebs]

@jeeyoungk - Thank you!

As a side note, as discussed in #3664, the AWS Go SDK used in Prometheus is very old, which might cause you guys some noise around EC2 discovery. Would be happy to see it upgraded.

[brian-brazil]

Pull requests welcome.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.