ec2_sd_config IAM roles support

[loky9000]

Is there are possibility to use AWS IAM roles in ec2_sd_config?
I test prometheus stack in docker (prom/prometheus:latest)
My aws user has no direct access to resources only through switch to correct IAM role so
i got
UnauthorizedOperation: You are not authorized to perform this operation.\n\tstatus code: 403
when access_key and secret_key params is set

i try set profile property (profile configured on docker host in ~/.aws/config so unclear how it can be used from inside prometheus container, if it should be mounted is unclear the target for this mount)

• job_name: 'aws_test'
  tls_config:
  insecure_skip_verify: true
  ec2_sd_configs:
  - region: 'eu-east-1'
  profile: "test-profile"
  relabel_configs:
  - source_labels: [__meta_ec2_tag_Name]
  regex: dmz-haproxy*
  action: keep
  # Use the instance ID as the instance label
  - source_labels: [__meta_ec2_instance_id]
  target_label: instance
  but got the error
  "Error loading config: couldn't load configuration (-config.file=/etc/prometheus/prometheus.yml): unknown fields in ec2_sd_config: profile" source="main.go:149"

[brian-brazil]

It should just work out of the box.

[loky9000]

Run successfully with profile: parameter after pull fresh prom/prometheus:latest
but there are couple issues after update:

• AWS nodes doesn't show in Targets(Looks like prometheus unable to get nodes from AWS using profile. Is there are any possibility to set role_arn (for example how it done for cloudwatch-exporter https://github.com/prometheus/cloudwatch_exporter)
• Expression browser doesn't show on /graph path anymore. 'Add graph' button do nothing.
  The same on v1.4.0 (works on v1.3.1)
  https://www.dropbox.com/s/tfsghsa7todiowh/Screenshot%202016-12-03%2011.48.50.png?dl=0
• Prometheus container run with '-log.level=debug' property but show log level info on /flags
  https://www.dropbox.com/s/anse9qrsva7r80j/Screenshot%202016-12-03%2011.55.45.png?dl=0

[loky9000]

So where is role_arn should be set?

[loky9000]

Looks like 'profile' parameter for ec2_sd_config not works or ignored for some reason
This my prometheus config:
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
evaluation_interval: 15s # By default, scrape targets every 15 seconds.
external_labels:
monitor: 'my-project'

rule_files:

• "alert.rules"

scrape_configs:

• job_name: 'aws_HAPROXY'
  ec2_sd_configs:
  • profile: 'devops'
    region: 'us-east-1'
    port: 9101
    refresh_interval: 10s
    tls_config:
    insecure_skip_verify: true
    relabel_configs:
  • source_labels: [__meta_ec2_tag_Name]
    regex: dmz-haproxy*
    action: keep
  • source_labels: [__meta_ec2_instance_id]
    target_label: instance

docker compose for prometheus looks like
services:
prometheus:
image: prom/prometheus:master
container_name: prometheus
volumes:
- ./prometheus/:/etc/prometheus/
- prometheus_data:/prometheus
- /etc/ssl/certs:/etc/ssl/certs:ro
- /etc/pki:/etc/pki:ro
- /root/.aws:/root/.aws:ro
command:
- '-config.file=/etc/prometheus/prometheus.yml'
- '-log.level=debug'
- '-storage.local.path=/prometheus'
- '-alertmanager.url=http://alertmanager:9093'
expose:
- 9090
ports:
- 9090:9090
links:
- cadvisor:cadvisor
- alertmanager:alertmanager
- cloudwatch-exporter:cloudwatch-exporter
depends_on:
- cadvisor
networks:
- back-tier
environment:
- "AWS_DEFAULT_REGION==us-east-1"
- "AWS_DEFAULT_PROFILE==devops"

AWS config files in /root/.aws folder:

cat ~/.aws/credentials
[default]
aws_access_key_id = XXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXX

cat ~/.aws/config
[default]
output = json
region = us-east-1

[profile devops]
role_arn = arn:aws:iam::xxxxxxxxxxxx:role/OmniDevOps
source_profile = default

Any logs about connection to aws on prometheus side. And hosts does not appear under /targets tab on prometheus UI.

[rlees85]

I also cannot get prometheus to work with an IAM role. This is the error I get from trying to do so on an ec2 instance with the correct IAM role attached to it:

time="2017-01-31T16:49:48Z" level=error msg="could not describe instances: EnvAccessKeyNotFound: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY not found in environment" source="ec2.go:81" 

It seems like it is not even trying because the access key value is not set.

A simple aws ec2 describe-instances works fine proving no issues with the IAM profile/role.

EDIT: This works in the latest prometheus. I did not check how old and crusty the version in the Ubuntu repositories were. Apologies!

[vacri]

@rlees85 what version is it working for you on? I'm using 1.5.0 from the debian/sid apt repo (built 20170126) and getting the EnvAccessKeyNotFound error, despite having a valid role on the instance. Just wondering if I need to build my own even newer version, or if it's something else.

[rlees85]

@vacri it was 1.5.2 from the Prometheus homepage: https://prometheus.io/download/

It was a case of manually extracting the file and finding a suitable systemd unit. I think I used this one:

https://askubuntu.com/questions/818514/how-do-i-install-the-latest-version-of-prometheus-on-16-04

(Last post in the thread, NOT the unit for node-exporter)

Make sure you change the path in ExecStart to match the version and path you extracted.

[vacri]

thanks @rlees85, I'll check it out!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.