Record seccomp profiles

[saschagrunert]

The idea is to allow the recording of seccomp profiles directly inside the Kubernetes cluster. To achieve that we could utilize the oci-seccomp-bpf-hook, which has the following requirements:

• Kernel headers have to be installed (could be handled by the operator or the sysadmin)
• An OCI hook compatible container runtime

We could think about annotating a node, then install the kernel headers (may be a prerequisite) and the hook. After that, the operator would have to annotate the target workload correctly (that's the way how the hook knows what to do). The recorded profile would have to be written into a configmap afterwards, which would cause that all nodes sync the profile to disk.

WDYT? It's overall a bit complicated but seems like a valuable feature after the initial release.

---

Edit: We could also use the seccomp notifier feature to think about a more elegant way to record profiles. This needs more evaluation but might be achievable by creating a PID namespace-shared sidecar.

[saschagrunert]

/priority important-longterm

[pjbgf]

This method is ideal for long running containers which would be "monitored" as its functionality is being used, as it only captures the syscalls made during the execution of the container being observed. The more permutations of execution paths ran, the more accurate will be the resulting profile.

A challenge on using seccomp-bpf-hook on a cluster is that it will need to run as root and with CAP_SYS_ADMIN capability, which effectively allows that container to bypass a considerable amount of security isolation primitives in the nodes. This can put end-users at risk if they were to decide using such feature in production or mixed usage clusters/nodes.

Ideally such approach would be used on ephemeral non-production clusters with the assistance of E2E tests which would be executed against the container being observed.

[saschagrunert]

Moving to v2.

[saschagrunert]

cc @rhafer @ereslibre @mrostecki

[moolen]

I did a bit of reasearch, i just want to cross-reference them:

• cri-containerd doesn't support oci-hooks yet, this is IMO a major blocker for this (Add support for runc hook specs containerd/containerd#6645)
• ftrace could be a different option (depends on oci-hooks, but doesn't require kernel headers): https://github.com/KentaTada/oci-ftrace-syscall-analyzer
• we could talk to containerd directly to get a PID and inject the eBPF tracer after the container starts (far from ideal, but currently possible)

[saschagrunert]

Interesting approaches @moolen and thank you for the insights. 🙏 From my perspective the most clean way would be to let containerd suppport OCI hooks, where I see no major blockers for it. 🤷

[rhafer]

A challenge on using seccomp-bpf-hook on a cluster is that it will need to run as root and with CAP_SYS_ADMIN capability, which effectively allows that container to bypass a considerable amount of security isolation primitives in the nodes. This can put end-users at risk if they were to decide using such feature in production or mixed usage clusters/nodes.

@pjbgf Isn't it only the hook itself that needs CAP_SYS_ADMIN (in order to setup the bpf tracer). IIUC this doesn't mean that the container itself needs the CAP_SYS_ADMIN capability.

[rhafer]

I did a bit of reasearch, i just want to cross-reference them:

• cri-containerd doesn't support oci-hooks yet, this is IMO a major blocker for this (containerd/containerd#6645)
• ftrace could be a different option (depends on oci-hooks, but doesn't require kernel headers): https://github.com/KentaTada/oci-ftrace-syscall-analyzer

Yeah, that might be worth a look. Another option would be to utilize the seccomp notifcation mechanism (as already mentioned in initial post), but that does require some changes on lower level runtimes (runc, crun, ...)

• we could talk to containerd directly to get a PID and inject the eBPF tracer after the container starts (far from ideal, but currently possible)

Hm, I guess at that point it might already have missed a couple of syscalls? So result would be imcomplete. I agree with @saschagrunert here, it would be cool if containerd would finally get support for oci-hooks.

[pjbgf]

@pjbgf Isn't it only the hook itself that needs CAP_SYS_ADMIN (in order to setup the bpf tracer). IIUC this doesn't mean that the container itself needs the CAP_SYS_ADMIN capability.

Very fair observation. Overall my concern is that requiring users to have something running with CAP_SYS_ADMIN capability within their clusters may lead a few potential route of privesc/security features bypass that cannot go unchecked. This is not a show stopper in itself, however we need to think what compensating controls we would put in place to limit potential damage.

Once we get to a working prototype we can brainstorm around it.

[saschagrunert]

/assign
I'll take over the work from @rhafer :)

[saschagrunert]

Let's close this for now, more enhancements to the feature to come.
/close

[k8s-ci-robot]

@saschagrunert: Closing this issue.

In response to this:

Let's close this for now, more enhancements to the feature to come.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.