[Fargate] [request]: Grant forensic capability by giving SYS_PTRACE for Fargate containers

[galaxy4public]

Tell us about your request
The biggest concern our organisation has that prevents us from moving our workloads to AWS Fargate is the lack of forensic capabilities on the platform. We asked our TAM about that and after consulting with AWS Fargate team they responded that it is not on the roadmap and that we should use 3rd party solutions.

We have capable developers and we could implement the required evidence gathering ourselves, however, to do so a process within the container should be able to do ptrace_attach().

The current set of capabilities provided for the containers running on Fargate is the following: chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap.

You may notice that the list is lacking ptrace, so there is no possibility to acquire core dumps inside the container to do post-mortems nor to do any tracing of the binaries.

Unless the SYS_PTRACE capability is provided to the containers running on Fargate there cannot be any "3rd party" or otherwise solutions to acquire forensic evidence from containers.

Which service(s) is this request for?
Fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

We need the ability to gather forensic evidence (memory dumps) in the event when an application in the container has been compromised.

Just imagine the following scenario: a container is running a vulnerable PHP application and is processing Personally Identifiable Information (e.g. medical records). An attacker has been able to successfully exploit a vulnerability and now eaves drop on all the data passing through the container.

The administrators of the system detected an unusual activity of a particular container and raised a question to the security team. At that point the normal flow would be to gather the forensic evidence to analyse whether it is a true positive or not and if it is, understand the root cause and the impact. To do so, we usually make a snapshot of the memory and a snapshot of the storage, so we could investigate -- followed by rolling back to the last known good state (to allow the system function until the investigation is under way).

With Fargate this is currently not possible due to lack of SYS_PTRACE. If it that capability was there, at least we could do gcore -o core <pid1> <pid2> ... to capture the memory dumps for the analysis.

Are you currently working around this issue?

We are currently advising the entire company not to use Fargate for any sensitive work loads since there is not support for forensics.

Additional context

It would be nice if the current decisions on the capabilities for Fargate containers were at least briefly documented. We extensively searched for any description why SYS_PTRACE was not included in Fargate, but all we could find is that you can add it to normal containers running on ECS, but that adding capabilities to Fargate containers is not supported.

[ejc3]

They did mention here they at least are considering exposing more capabilities:
https://twitter.com/nathankpeck/status/1122188790452371456

[ejc3]

Just curious if the prohibition at your company extends to Lambda as well.

[galaxy4public]

Just curious if the prohibition at your company extends to Lambda as well.

For starters it is not a "prohibition" but a strong "discouragement" to use. :) Anyway, with Lambda we currently a bit softer on it since Lambdas are supposedly short-lived and it is really hard to come up with a use-case when we would need to analyse the memory inside a Lambda container space. This also requires stricter framework on the input sanitization and validation for Lambdas since it is the only feasible attack surface there.

With Fargate, on the other hand: the containers are long-lived most of the time and often provide customer-facing services -- i.e. directly interacting with untrusted user input, so the likelihood of a vulnerability exploitation is significantly higher.

[akshayram-wolverine]

Shipped! Details here: https://aws.amazon.com/about-aws/whats-new/2020/04/aws-fargate-launches-platform-version-14/

[mlanner-aws]

Closing this issue as we've shipped the feature.