Expose AWS credentials on a per-task basis

[egoldschmidt]

Instance role credentials, exposed via the instance metadata service, are a great way to provide tightly-scoped access to AWS resources on a per-service basis. Unfortunately, ECS does not play nicely with instance profiles. There is no ability, as far as I know, to provide a runtime role on a per task, task definition, or service basis, so all tasks running on the same instance must act as the same role.

Supporting instance roles on a per container basis seems inevitable, whether exposed via the agent or future mystical means. I haven't found an existing issue so I figured I'd get the ball rolling.

A few ideas:

• A custom Docker networking driver could be used, perhaps, to route instance metadata requests from the container back to the ECS agent, which could use STS to mint credentials based on the originating container.
• Containers could be launched with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables prepopulated.

Of course there are security implications to exposing credentials in either of these ways, but I can't see how it could be worse than running instances with the superset of all possible permissions.

[aaithal]

Hi @egoldschmidt, thanks for the feedback. I have captured your suggestion as a feature request and will update this issue when there is additional information.

[maclennann]

While a native solution would certainly be better, in the meantime I wanted to call out metadataproxy from lyft as a potential stop-gap.

Your ECS container instance runs under an instance profile that can assume all of the roles that could possibly be required by a container, then any container can specify an IAM_ROLE env-var. With the addition of a simple iptables rule, all requests to the metadata service can be proxied through this container and get STS credentials scoped appropriately for the container.

Of course, this doesn't protect against maliciousness (e.g. containers can still try to assume a SUPER_ADMIN role and the proxy will try to help them out), but it's a good safeguard and sanity-check.

Note that I haven't tried this in production yet so I can't vouch for it personally.

[jhovell]

There's also https://github.com/dump247/ec2metaproxy which I think was possibly the inspiration for Lyft's metadataproxy. I haven't tried either yet, but hoping support for this is coming soon from AWS.

[ghost]

I wish ECS could find a way to assign a network interface for each container. If it were possible, a container could retrieve metadata just as an EC2 instance can. The metadata could be scoped to whatever role we wished to assign the container in the ECS Task. It would have many other advantages too, as a container would become a first class citizen of the VPC's network.

I have seen there are limits on the number of network interfaces an EC2 instance can have though, so I won't hold my breath for this.

[ProTip]

There are quite a few ways this could be solved without needing and ENI per container. For instance, because the ECS agent knows what virtual interface each container is configured on it can setup custom EB/IP-tables rules to handle sending back custom information in any number of ways.

Looks like this has been addressed though. I wish "pending release" provided a bit more insight. Still though, this issue tracker is the greatest visibility into ECS plans I have seen. When I spoke to them at Docker Con SF last year they were already aware of this use-case however I have heard ZERO about their future plans for it or ECS since :|

[jefferai]

I'm in the same boat as @yeungda-rea -- real, signed, unique-per-container metadata would be immensely useful for container identification and authorization.

[kiranmeduri]

@jefferai We are interested in learning more about your use-case. Are you looking for something in the lines of EC2 Instance Metadata http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html? Today that endpoint is not signed but only provides data to processes on that instance. It seems like you prefer a TLS enabled endpoint accessible only to the containers within a particular task. Am I understanding it correctly? Also it will be great if you can explain your use-case of identification and authorization.

Thanks for the feedback.

[jefferai]

Hi @kiranmeduri,

Sure, I'm happy to go into details.

In Vault we have an authentication backend that uses Amazon as a Trusted Third Party to enable automatic authentication of EC2 instances. You can find the docs here, but I'll give a very high level discussion of the flow.

The basic idea revolves around two facts: one, that Amazon publishes a signed PKCS#7 version of the instance metadata, and the signature can be verified with Amazon's published key; two, the principle of Trust On First Use (TOFU).

When an instance (with our agent, or another's) starts up, one of the first tasks it does is fetch the PKCS#7 signed document and send it over to the Vault server. The Vault server first validates the signature/HMAC on the document to ensure that the document was signed by AWS and not modified, so that we can trust the contents (if we trust AWS, which we do). One thing to keep in mind here is that since this signed metadata is only available within the EC2 instance, it means that something sending over this signed metadata must be running on the instance (unless it leaks over time -- but that's why it's TOFU). You can't simply look up instance IDs in the external API and construct a document yourself, because you don't have the AWS signing key.

Vault then verifies that the instance ID has not authenticated before. If so, authentication is disallowed, and a legitimate client can raise an alarm indicating that some other process has already used this metadata to authenticate.

Otherwise, Vault uses the EC2 API to check that the instance is indeed alive and running, and also can verify a number of other optional attributes (account ID, subnet ID, etc.). If everything matches, an authentication token is returned to the caller, and the fact that the instance has authenticated is recorded.

It's a bit more complicated than this but overall by ensuring that a legitimate client can authenticate, while also detecting if a bad actor has managed to authenticate in its stead, it provides reasonably good security guarantees -- certainly good enough for some very security-conscious users of ours that helped design it.

Unfortunately, I've been around and around the ECS API both from within and without of the container and there's just not enough unique information that can be correlated across them. Even if there were, part of the reason the above process works is that there is information that we can verify (using the AWS public key) but that is only available inside the instance. As an example, exposing the Docker IDs to the external ECS API wouldn't help, because although it uniquely identifies a container and could be correlated both within and without (using a task ID alongside it), this also means anyone with access to the external API could spoof login requests -- too easy/risky.

In theory we could fork the ECS agent so that the ECS agent could authenticate the EC2 instance to Vault and then dish out tokens to containers that are starting up, but we'd rather not require users to run forked versions of the agent (and don't think most of our users would).

So for our use case, signed metadata available within the container containing uniquely identifying information that can be corroborated using the external API would allow us to use the same mechanism for ECS containers. So far our users are very happy with the EC2 functionality and its security stance and tradeoffs, so they'd probably be equally happy with the same solution inside of ECS. But it does require more than simply exposing a couple of new values externally and/or internally -- the signed metadata from AWS is key.

[aaithal]

We have released the Task IAM Roles for ECS today, which should address the issue of vending IAM Role Credentials to containers.

@jefferai thanks for providing those details about your use-case. It seems like you would benefit from some sort of ECS-mediated container identity solution. However, IAM Roles Credentials for containers is not intended to solve that use-case. I have created a #451 to track that feature request. We really appreciate the detail that you provided.

Thanks,
Anirudh

[jhovell]

congrats and awesome! Did I miss this at the keynote at the summit today?? this is huge, can't wait to try it out...

[dinvlad]

Bringing up the issue topic once again, task roles still don't grant per-container permissions. Are there future plans to add support for these?

It seems that since all containers within a single task definition have to share the same permissions, we cannot really use multi-container task functionality in sensitive scenarios. That severely limits some applications, e.g. if we want to pair each "client-facing" container with a "co-pilot service" container on the same host (i.e. not as an actual ECS service, but rather unique for each client), we have to split them in separate tasks and write our own scheduler that would ensure that both tasks are scheduled to the same instance and are given enough resources to accommodate both.

Of course, there are workarounds to that (e.g. running a single service on each host that serves multiple clients). Still, they introduce an additional layer of complexity.

It seems that adding "container role arn" as an option for "container overrides" would address this problem without changing the current API.

[richardpen]

@dinvlad This seems like another feature request, could you create another issue for us to track, thanks.