[Need help] Can't run the first docker command

[ozbillwang]

I have followed the Usage README, set sts:assume-role and trust Relationship to be assumed by the root role.

I currently login the ec2 instance which has the iam role with above changes.

But when I run the first docker command (docker run --volume /var/run/docker.sock:/var/run/docker.sock --restart=always --net=host swipely/iam-docker:latest), I got below error:

2018-07-02T05:14:59Z [docker] Starting event handler event-handler=1
2018-07-02T05:14:59Z [docker] Starting event handler event-handler=2
2018-07-02T05:14:59Z [docker] Starting event handler event-handler=3
2018-07-02T05:14:59Z [docker] Adding container event-handler=4 id=d50238ea7d161e5ed89844f638d7e10cf0750049678c237c1c11bc86640c2337 event=start
2018-07-02T05:14:59Z [docker] Unable to add container event-handler=4 id=d50238ea7d161e5ed89844f638d7e10cf0750049678c237c1c11bc86640c2337 event=start error="Unable to find label named 'com.swipely.iam-docker.iam-profile' or environment variable 'IAM_ROLE' for container: d50238ea7d161e5ed89844f638d7e10cf0750049678c237c1c11bc86640c2337"
panic: runtime error: index out of range

error="Unable to find label named 'com.swipely.iam-docker.iam-profile' or environment variable 'IAM_ROLE' for container:

then I feed environment variable iam-profile into container:

export PROFILE="arn:aws:iam::1234567:role/iam_role_dev "

docker run -e IAM_ROLE="$PROFILE"  --volume /var/run/docker.sock:/var/run/docker.sock --restart=always --net=host swipely/iam-docker:latest

I got another error

2018-07-02T05:26:01Z [docker] Adding container event-handler=4 id=2fd5a4a2aa09dd0cf9d733bc8977c718929dc5247d60a332c5468cf9141d3534 event=start
2018-07-02T05:26:01Z [docker] Unable to add container id=2fd5a4a2aa09dd0cf9d733bc8977c718929dc5247d60a332c5468cf9141d3534 event=start event-handler=4 error="Unable to find IP address for container: 2fd5a4a2aa09dd0cf9d733bc8977c718929dc5247d60a332c5468cf9141d3534"
panic: runtime error: index out of range

error="Unable to find IP address for container

how to fix the issue?

I already run the docker command with root permission.

Conslution

1. Only image swipely/iam-docker:v1.0.0 works in my enviroinment. Tags with latest, v1.2.0 and v1.1.0 doesn't work.
2. Make sure you set the assume role on both (instance iam role and the role which container will be assumed)
3. Make sure port 8080 is not used by the host, otherwise, use other ports (reference: [Need help] Can't run the first docker command #25 (comment))

[ozbillwang]

Seems the error is from src/docker/container_store.go

        if len(ips) == 0 {
                return nil, fmt.Errorf("Unable to find IP address for container: %s", id)
        }

We have firewall applied to ec2 instance, should we open the port 8080 which is not list in the inbound rules?

[nahiluhmot]

You should not need to open up port 8080, all of the requests are internal relative to the instance.

I see that you're setting IAM_ROLE on the iam-docker container. You should instead be setting that environment variable on each container that you run.

Also, can you confirm that you ran the iptables setup instructions from the README?

[ozbillwang]

So I need run iptables command before docker run? I will try it.

Thanks.

[nahiluhmot]

Correct.

[ozbillwang]

@nahiluhmot

I have added the REDIRECT

$ iptables -t nat -L --line-numbers |grep instance-data
2    REDIRECT   tcp  --  anywhere             instance-data.xxx.compute.internal  tcp dpt:http redir ports 8080

$ nslookup instance-data.xxx.compute.internal
Non-authoritative answer:
Name:	instance-data.xxx.compute.internal
Address: 169.254.169.254

But still get same error when docker run.

[ozbillwang]

Seems my problem is closed to this issue #9, port 8080 is used already. I change to 8888

$ iptables -t nat -I PREROUTING  -p tcp -d 169.254.169.254 --dport 80 -j DNAT --to-destination "172.17.0.1:8888" -i "docker0"

$ docker run --volume /var/run/docker.sock:/var/run/docker.sock --restart=always --net=host swipely/iam-docker:latest /iam-docker -listen-addr=0.0.0.0:8888 -verbose

Now I got the same error:

Unable to find label named 'com.swipely.iam-docker.iam-profile' or environment variable 'IAM_ROLE' for container

[ozbillwang]

@nahiluhmot

Seems the issue is a bug about some new commits between v1.0.0 and v1.2.0

I did tests with v1.2.0. v1.1.0 and v1.0.0, it runs fine with v1.0.0 only

curl 172.17.0.1:8888
1.0
2007-01-19
2007-03-01
2007-08-29

172.17.0.1 is the gateway IP.

Now I start testing the real function, will update for the result.

Update 1

Got below error.

$ docker run -ti --rm --label com.swipely.iam-docker.iam-profile="arn:aws:iam::xxxx:role/s3-readonly" fstab/aws-cli sh
$ /home/aws/aws/env/bin/aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".

And error login container iam-docker

2018-07-03T06:44:25Z [app] Refreshing credentials worker=refresh-credentials
2018-07-03T06:44:25Z [iam] Refreshing all IAM credentials
2018-07-03T06:44:25Z [iam] Done refreshing all IAM credentials
2018-07-03T06:44:28Z [http] Delegating request upstream remoteAddr=172.17.0.2:34638 path=/latest/meta-data/iam/security-credentials method=GET
2018-07-03T06:44:28Z [http] Serving list IAM credentials request method=GET remoteAddr=172.17.0.2:34638 path=/latest/meta-data/iam/security-credentials/
2018-07-03T06:44:28Z [docker] Looking up IAM role ip=172.17.0.2
2018-07-03T06:44:28Z [http] Unable to find credentials path=/latest/meta-data/iam/security-credentials/ method=GET remoteAddr=172.17.0.2:34638 error="Unable to find container for IP: 172.17.0.2"

I can get the iam role name when run curl command on its host

 $ curl 169.254.169.254/latest/meta-data/iam/security-credentials/

But when run the same command in container, I got 404 error

Update 2

The gateway is 172.17.0.1, but the error log in container iam-docker is pointed to 172.17.0.2, that's why it can't get host iam role's security-credentials

If I manually curl to 172.17.0.1 in container, it works.

Update 3

I am very close to make it work, in fact the error in update 2 can be ignored, then I found the real error which point to the right direction.

2018-07-03T07:21:53Z [http] Unable to find credentials path=/latest/meta-data/iam/security-credentials/ method=GET remoteAddr=172.17.0.4:32856 error="AccessDenied: User: arn:aws:sts::xxx:assumed-role/dev_iam_role/i-0a9099bcxxxx is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxx:role/s3-readonly\n\tstatus code: 403, request id: c2c220c1-7e91-11e8-bb64-xxx"

But I have applied below trust relationships policy to the host's iam role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxx:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

[ozbillwang]

I finally made it work.

The missing part is, I didn't set properly for Setup an instance IAM role that can perform sts:assume-role on the roles you'd like to assume. in the README

So Make sure you set the assume role on both (instance iam role and the role which container will be assumed)

reference Netflix/security_monkey#732 (comment)