AWS Feature: Integration with IAM

[justinsb]

It would be great to be able to configure IAM permissions on a per-pod basis. We would then have a "mock" metadata service on 169.254.169.254, which would inject the correct IAM roles into the pod (or no roles at all).

I believe this can be implemented using the AWS Security Token Service:
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html

[justinsb]

See also #14226

[guoshimin]

We here at Databricks is doing something similar to https://github.com/dump247/ec2metaproxy:

• We run a metadata proxy on each node, using DaemonSet
• The proxy uses host network namespace
• We DNAT calls to 169.254.169.254 in the PREROUTING chain to the proxy
• The proxy checks the source IP against the pod list to identify the pod
• Pods that need a role specify that in their annotations
• The proxy calls sts::AssumeRole to get the temporary crendentials for the role and forwards these to the requesting container.
• An authorization plugin enforces policies on what pods can use what roles

As @therc pointed out in this comment, it doesn't work for containers that use net=host, but those containers tend to perform admin functions and tend to be static, so they can use the role associated with the node itself.

[therc]

@guoshimin did you say it's written in Scala?

[guoshimin]

@therc It is indeed written in Scala.

[mgoodness]

Lyft recently released a similar project to ec2metaproxy, called metadataproxy. Announcement here. Haven't tried it myself, but might be worth investigating.

[ghost]

cc @erictune FYI.
Do we need an area/iam label?

[gtaylor]

@guoshimin Is your scala implementation specifically for Kubernetes? If so, I don't suppose that it's open source? Keen to see how other people are tackling this in production.

[thomasdesr]

@gtaylor Shimin's co-worker here

Ours is specific for kubernetes as it relies on pod annotations to determine the appropriate role. Sadly it isn't open source, however @jtblin did write something very simliar and did open source it (https://github.com/jtblin/kube2iam) :D

Our internal one differs from kube2iam only in automatically running the needed iptables command if it isn't already set and requiring about 10-100x as much memory :P

Minor edit: Our AWS Role permissions are slightly more restricted than the examples in kube2iam, for example our minions don't have arbitrary assume-role permissions, they are only allowed to assume other roles that have established a trust relationship with them (that are inside the same account, cross-account isn't a problem we've needed to solve yet).

[jtblin]

Author of https://github.com/jtblin/kube2iam, we've been using it in our environment for quite some time without issue. We also use a slightly more restricted set of permissions than the example in the README, we only give sts:AssumeRole to a specific role instead of root. I shall update the README.

The code is inspired from the original kube2sky, using similar APIs.

[evie404]

Question regarding sts:AssumeRole and running proxies with DaemonSet:

It seems while we can use trust relationship to limit the number of roles the Kubnernetes workers are allowed to assume, it still means that Kubernetes workers, which runs all applications as Pods, can potentially expose all roles and is a single point of failure.

Instead, would it be possible that, instead of running the proxies with DaemonSet, run them as standalone instances, say iam-proxy, and route the Pods' 169.254.169.254 calls to them? This at least prevent containers with privileged access potentially gaining any roles.

[erictune]

But a proxy would be single point of failure too. If it goes down, pods cannot refresh their creds via 169.254.169.254.

[erictune]

It depends on whether you want better uptime or better security.

[evie404]

I was thinking running multiple-instances of proxies behind a load balancer to provide redundancy. But at least this way, the proxies' roles are isolated.

[yissachar]

Is this planned for 1.4? Trying to decide whether to wait for this to be implemented or just go ahead and use https://github.com/jtblin/kube2iam.

[erictune]

Not planned for 1.4.

On Wed, Aug 17, 2016 at 12:47 PM, yissachar notifications@github.com
wrote:

Is this planned for 1.4? Trying to decide whether to wait for this to be
implemented or just go ahead and use https://github.com/jtblin/kube2iam.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#23580 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AHuudnOc5LrihsMRfDvpN6vaNKxtXEXZks5qg2VTgaJpZM4H61a1
.

[erictune]

Looks like there are several of these:

1. Lyft
   https://github.com/lyft/metadataproxy
   https://eng.lyft.com/scoping-aws-iam-roles-to-docker-containers-c9c5f8f2f75#.2svjixfff

2. jtblin (integrates with Kube)
   https://github.com/jtblin/kube2iam

3. swipely
   https://github.com/swipely/iam-docker

4. dump247
   https://github.com/dump247/docker-ec2-metadata

[evie404]

the docker iam proxies all assume the docker networking model of one IP per container. since they use IP to look up the associated containers, they will not work. kube2iam by contrast looks through the kubernetes API, so does not have this shortcoming. also, it should support rkt out of the box since it does not have dependency on the docker networking model nor the docker socket.

see: #14226 (comment)

[Vlaaaaaaad]

Any updates on this? What's the preferred way to grant permissions at a pod/container level on AWS?

[garrickpeterson-wf]

Also interested in this. When you want this it's one of those cases, to address earlier comments, where security trumps availability (though, couldn't availability be addressed via daemonsets or similar?).

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close