Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix AWS IAM Roles for Service Accounts permission problem. #1185

Conversation

@serialx
Copy link
Contributor

serialx commented Sep 11, 2019

Amazon EKS supports IAM Roles for Service Accounts. It mounts tokens files to /var/run/secrets/eks.amazonaws.com/serviceaccount/token. Unfortunately, external-dns runs as 'nobody' so it cannot access this file. External DNS is then unable to make any AWS API calls to work:

time="2019-09-11T07:31:53Z" level=error msg="WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied"

See: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

Below are the file permissions mounted on External DNS pod:

~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/
total 0
drwxrwxrwt    3 root     root           100 Sep 11 06:40 .
drwxr-xr-x    3 root     root            28 Sep 11 06:40 ..
drwxr-xr-x    2 root     root            60 Sep 11 06:40 ..2019_09_11_06_40_49.865776187
lrwxrwxrwx    1 root     root            31 Sep 11 06:40 ..data -> ..2019_09_11_06_40_49.865776187
lrwxrwxrwx    1 root     root            12 Sep 11 06:40 token -> ..data/token
~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
-rw-------    1 root     root          1028 Sep 11 06:40 /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token

Pinging @taharah for review.

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Sep 11, 2019

Welcome @serialx!

It looks like this is your first PR to kubernetes-incubator/external-dns 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-incubator/external-dns has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot requested review from hjacobs and linki Sep 11, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Sep 11, 2019

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.


Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@serialx

This comment has been minimized.

Copy link
Contributor Author

serialx commented Sep 11, 2019

Updated CLA account settings.

@mikkeloscar

This comment has been minimized.

Copy link
Contributor

mikkeloscar commented Sep 11, 2019

Interesting. Do you know how the file permissions are for the Kubernetes service account token? Because this is mounted in a similar way and works with the nobody user.

@serialx

This comment has been minimized.

Copy link
Contributor Author

serialx commented Sep 11, 2019

Kubernetes tokens seems to be o+r

/run/secrets/kubernetes.io/serviceaccount $ ls -al
total 0
drwxrwxrwt    3 root     root           140 Sep 11 07:31 .
drwxr-xr-x    3 root     root            28 Sep 11 07:31 ..
drwxr-xr-x    2 root     root           100 Sep 11 07:31 ..2019_09_11_07_31_49.116680770
lrwxrwxrwx    1 root     root            31 Sep 11 07:31 ..data -> ..2019_09_11_07_31_49.116680770
lrwxrwxrwx    1 root     root            13 Sep 11 07:31 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root            16 Sep 11 07:31 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root            12 Sep 11 07:31 token -> ..data/token
/run/secrets/kubernetes.io/serviceaccount $ ls -al ..data/
total 12
drwxr-xr-x    2 root     root           100 Sep 11 07:31 .
drwxrwxrwt    3 root     root           140 Sep 11 07:31 ..
-rw-r--r--    1 root     root          1025 Sep 11 07:31 ca.crt
-rw-r--r--    1 root     root            12 Sep 11 07:31 namespace
-rw-r--r--    1 root     root           875 Sep 11 07:31 token
@mikkeloscar

This comment has been minimized.

Copy link
Contributor

mikkeloscar commented Sep 11, 2019

Very interesting, this means containers must run as root to use this feature. I don't think this is intended and we should make at least AWS aware.

@serialx

This comment has been minimized.

Copy link
Contributor Author

serialx commented Sep 11, 2019

Reported to the amazon-eks-pod-identity-webhook repository.

@devkid

This comment has been minimized.

Copy link
Contributor

devkid commented Sep 11, 2019

@serialx

This comment has been minimized.

Copy link
Contributor Author

serialx commented Sep 11, 2019

Damn. But both code uses ServiceAccountToken. Maybe the code path is different?

@devkid

This comment has been minimized.

Copy link
Contributor

devkid commented Sep 11, 2019

No, this one is for projected service account tokens.

@devkid

This comment has been minimized.

Copy link
Contributor

devkid commented Sep 11, 2019

@linki

This comment has been minimized.

Copy link
Contributor

linki commented Sep 11, 2019

I was able to read a projected service account token with nobody by using securityContext.fsGroup:

      ...
      securityContext:
        fsGroup: 65534
      volumes:
      - name: token-vol
        projected:
          sources:
          - serviceAccountToken:
              path: token
$ whoami
nobody
$ ls -la /var/run/secrets/kubernetes.io/serviceaccount/..data/
-rw-r-----    1 root     nobody        1102 Sep 11 15:33 token
@marcincuber

This comment has been minimized.

Copy link

marcincuber commented Sep 13, 2019

Any chances to get a timeline when this is going to be released?

@Raffo

This comment has been minimized.

Copy link
Contributor

Raffo commented Sep 13, 2019

I'd defer to @linki , but I think that if a change of configuration will allow to do that, I'd rather change the docs to mention this than change the dockerfile.

@jqmichael

This comment has been minimized.

Copy link

jqmichael commented Sep 14, 2019

65534

https://stackoverflow.com/questions/34831861/can-i-assume-that-nobody-is-65534

You can't. nobody has had at least a few different IDs across distros and time:

Historically, the user “nobody” was assigned UID -2 by several operating systems, although other values such as 2^(15)−1 = 32,767 are also in use, such as by OpenBSD. For compatibility between 16-bit and 32-bit UIDs, many Linux distributions now set it to be 2^(16)−2 = 65,534; the Linux kernel defaults to returning this value when a 32-bit UID does not fit into the return value of the 16-bit system calls. An alternative convention assigns the last UID of the range statically allocated for system use (0-99) to nobody: 99.

@serialx

This comment has been minimized.

Copy link
Contributor Author

serialx commented Sep 16, 2019

Yeah, but the docker file already specifies 65534, so it doesn't really matter does it?

https://github.com/kubernetes-incubator/external-dns/blob/master/Dockerfile#L36

Amazon EKS supports IAM Roles for Service Accounts. It mounts tokens
files to `/var/run/secrets/eks.amazonaws.com/serviceaccount/token`.
Unfortunately, external-dns runs as 'nobody' so it cannot access this
file. External DNS is then unable to make any AWS API calls to work:

```
time="2019-09-11T07:31:53Z" level=error msg="WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token\ncaused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied"
```

See: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

Below are the file permissions mounted on External DNS pod:

```
~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/
total 0
drwxrwxrwt    3 root     root           100 Sep 11 06:40 .
drwxr-xr-x    3 root     root            28 Sep 11 06:40 ..
drwxr-xr-x    2 root     root            60 Sep 11 06:40 ..2019_09_11_06_40_49.865776187
lrwxrwxrwx    1 root     root            31 Sep 11 06:40 ..data -> ..2019_09_11_06_40_49.865776187
lrwxrwxrwx    1 root     root            12 Sep 11 06:40 token -> ..data/token
~ $ ls -al /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
-rw-------    1 root     root          1028 Sep 11 06:40 /var/run/secrets/eks.amazonaws.com/serviceaccount/..data/token
```

This commit fixes this problem by specifying securityContext to make
mounted volumes with 65534 (nobody) group ownership.
@serialx serialx force-pushed the serialx:fix-amazon-iam-roles-service-account branch from 729557a to c97781a Sep 16, 2019
@serialx

This comment has been minimized.

Copy link
Contributor Author

serialx commented Sep 16, 2019

Can confirm workaround by using securityContext. I've changed the PR to update AWS tutorial instead of changing the Dockerfile. Should work fine with future k8s updates too.

@linki

This comment has been minimized.

Copy link
Contributor

linki commented Sep 16, 2019

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Sep 16, 2019
@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Sep 16, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: linki

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 894360a into kubernetes-sigs:master Sep 16, 2019
2 of 3 checks passed
2 of 3 checks passed
tide Not mergeable. Needs approved, lgtm labels.
Details
cla/linuxfoundation serialx authorized
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@farshad-hobsons farshad-hobsons mentioned this pull request Oct 9, 2019
3 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.