[Fargate] [request]: Provide the ability to use ebpf on fargate instances.

[KnoxAnderson]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
Provide the ability to leverage ebpf for security and monitoring use cases on fargate

Which service(s) is this request for?
ECS or EKS running on fargate

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

An eBPF program is "attached" to a designated code path in the kernel. When the code path is traversed, any attached eBPF programs are executed. Given its origin, eBPF is especially suited to writing network programs and it's possible to write programs that attach to a network socket to filter traffic, to classify traffic, and to run network classifier actions.

We'd want to attach eBPF programs to the following static tracepoints:

• System call entry path
• System call exit path
• Process context switch
• Process termination
• Minor and major page faults
• Process signal delivery

This allows the collection of -

• Data associated to a network connection (e.g. TCP/UDP IPv4/IPv6 tuple, UNIX socket names, …).
• Highly granular metrics about the process (memory counters, page faults, socket queue length, …).
• Container-specific data, such as the cgroups the process issuing the system call belongs to, as well as the namespaces that process lives in.

Are you currently working around this issue?
We are currently working around this issue by using ptrace which was exposed in fargate 1.4, but ebpf would be a more stable cross platform approach.

Additional context
Anything else we should know?

Attachments
If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[thomasdullien]

On our end: We are using eBPF to both collect performance data, but also for general metrics / observability and debugging mystery issues that arise. Proper support for eBPF in fargate would be great!

[bhiggins]

We'd really like to see uprobe attachment supported as well. We instrument processes using eBPF attached to uprobes and we'd love to see this work with Fargate!

[brendangregg]

I've been asked a number of times to provide tests to show that eBPF observability works in a given environment, including for containers. Here is my smallest set of tests (each testing something different):

bpftrace -e 'BEGIN { printf("hello world\n"); }'
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'
bpftrace -e 'kprobe:vfs_read { @start[tid] = nsecs; @bsize[arg2]++; } kretprobe:vfs_read /@start[tid]/ { @ns[comm] = hist(nsecs - @start[tid]); delete(@start[tid]); }'
bpftrace -e 'ur:/bin/bash:readline { printf("got: %s\n", str(retval)); }'
bpftrace -e 'profile:hz:9 { @[kstack, ustack] = count(); }'

For more tests, I'd try the tools in https://github.com/iovisor/bcc and https://github.com/iovisor/bpftrace. Note that the bcc tools are evolving into the versions in the libbpf-tools directory, which produce C binaries with no dependencies (no LLVM) as they contain embedded BPF bytecode. Also note that I'm discussing observability only here.

Making everything work in containers securely (and, once the bpf and perf_event_open syscalls are allowed, avoiding memory reads of other container info) will be quite some work.

[soo-o]

Hi folks, please pardon the extended delay in response for this issue. This is something we're actively working on (we've updated the status to "Coming Soon" rather than "Proposed"). In an incoming release, Fargate will make it possible to add specific Linux capabilities to the task containers, including CAP_BPF and CAP_PERFMON. Fargate will exclusively support BPF CO-RE applications.

[inge4pres]

In an incoming release, Fargate will make it possible to add specific Linux capabilities to the task containers, including CAP_BPF and CAP_PERFMON.

Thanks for the update @soo-o 🙏🏼
Do you already know the full list of capabilities that will be allowed?

I believe that since #1000 stays open, and given the security considerations on Fargate, CAP_SYS_ADMIN is off the table?

[NyanHelsing]

I'd be super stoked to see this functionality landed

[fntlnz]

@soo-o just saw that this is coming - can't wait! If y'all need someone to test it at some point I'm down for it.