SECCOMP events are not logged by auditd?

[igcherkaev]

Description

Unable to get auditd log SECCOMP violations with Flarcar Linux's auditd.

Impact

We're enabling default runtime seccomp profile by default on our kubernetes workers running on Flatcar Linux. It looks like it's working, but to collect potential impact from it we're trying to audit kernel events and capture SECCOMP messages, but no dice.

Environment and steps to reproduce

1. Set-up:

# cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3815.2.1
VERSION_ID=3815.2.1
BUILD_ID=2024-03-17-2158
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3815.2.1 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3815.2.1:*:*:*:*:*:*:*"

# uname -a
Linux server1 6.1.81-flatcar #1 SMP PREEMPT_DYNAMIC Sun Mar 17 21:32:50 -00 2024 x86_64 Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz GenuineIntel GNU/Linux

# containerd --version
containerd github.com/containerd/containerd v1.7.13 99b8088b873ba42b788f29ccd0dc26ebb6952f1e

# sysctl -a | grep seccomp
kernel.seccomp.actions_avail = kill_process kill_thread trap errno user_notif trace log allow
kernel.seccomp.actions_logged = kill_process kill_thread trap errno user_notif trace log

2. Task: audit seccomp-related messages
3. Action(s):
   3.1. Enable auditd service as described at https://www.flatcar.org/docs/latest/setup/security/audit/
   3.2. Remove 99-default.rules from /etc/audit/rules.d
   3.3. Add a custom rule:

-a exclude,never -F msgtype=SECCOMP

3.4. Reload auditd/reboot
3.5. Start a container with security context defined as:

    securityContext:
      seccompProfile:
        type: "RuntimeDefault"

3.6. Exec to the container and run:

$ kubectl -n kube-system exec -ti ubuntu -- bash
root@ubuntu:/# unshare
unshare: unshare failed: Operation not permitted
root@ubuntu:/# 

4. Error: Check /var/log/audit/audit.log on the host and there's no SECCOMP message.

Expected behavior

Expected to see type=SECCOMP lines in the audit log, but none are seen.

Additional information

I tried all kind of rules, but never seen any events logged with type=SECCOMP or 1326.
Is it me doing something wrong or containerd enforces seccomp policies somehow differently?

Thank you, and sorry if this is not Flatcar Linux's problem.

[tormath1]

Hello, I think it's more a configuration issue. The default containerd seccomp profile do not log by default (https://github.com/containerd/containerd/blob/3b0b3e533ce1f60a60ab83905374d1ce43330063/contrib/seccomp/seccomp_default.go#L485).
If you are just interested by logging first, you can have a look to this: https://kubernetes.io/docs/tutorials/security/seccomp/#create-a-pod-with-a-seccomp-profile-for-syscall-auditing (and then confirm that you do see the SECCOMP logs).

You can quickly test with:

core@localhost ~ $ cat audit.json
{
    "defaultAction": "SCMP_ACT_LOG"
}
core@localhost ~ $ docker run --rm -it --security-opt seccomp="$PWD/audit.json" debian:jessie unshare --map-root-user --user sh -c whoami
core@localhost ~ $ sudo cat /var/log/audit/audit.log | grep unshare
type=SECCOMP msg=audit(1711359381.608:980): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=system_u:system_r:container_t:s0:c782,c881 pid=2356 comm="unshare" exe="/usr/bin/unshare" sig=0 arch=c000003e syscall=59 compat=0 ip=0x7fa9722dc317 code=0x7ffc0000AUID="unset" UID="root" GID="root" ARCH=x86_64 SYSCALL=execve

[igcherkaev]

@tormath1 Thanks for the suggestion! I was under impression that ActErrno should also be logged if configured with the appropriate sysctl parameter?

https://elixir.bootlin.com/linux/v6.1.18/source/kernel/seccomp.c#L1221 it goes to skip at https://elixir.bootlin.com/linux/v6.1.18/source/kernel/seccomp.c#L1312 - which conditionally logs. From there it's little hard to understand how that match->log is computed/figured out, I assumed it was based on the sysctl parameter kernel.seccomp.actions_logged, but I could be wrong. Thought to seek help from the people who maintain OS and know better.

My goal here is enable seccomp default runtime profile and collect evidence with auditd that certain workloads break because of seccomp and not something else. My guess is if I make default action SCMP_ACT_LOG, it'll allow everything, but log it. Not exactly what we need here.

[tormath1]

I think it's a bit like SELinux: you first need to run SELinux in a permissive mode to ensure that you don't have any denials in your logs then you can enforce SELinux.
Here, if you want to be sure that RuntimeDefault won't cause any issue, you can apply its variant with "defaultAction": "SCMP_ACT_LOG" and you'll see breakage from the profile (as mentioned in the Kubernetes documentation)

[igcherkaev]

I think it's a bit like SELinux: you first need to run SELinux in a permissive mode to ensure that you don't have any denials in your logs then you can enforce SELinux.
Here, if you want to be sure that RuntimeDefault won't cause any issue, you can apply its variant with "defaultAction": "SCMP_ACT_LOG" and you'll see breakage from the profile (as mentioned in the Kubernetes documentation)

I get it, it makes sense when you introduce it for the first time to the cluster or a set of servers and you want to make sure nothing breaks. However, in the long run, it doesn't make sense not to be able to log it, as it is essential to understand what is preventing workloads from doing what they need/want to do.

I believe the following sysctl parameters were designed for exactly that:

# sysctl -a | grep seccomp
kernel.seccomp.actions_avail = kill_process kill_thread trap errno user_notif trace log allow
kernel.seccomp.actions_logged = kill_process kill_thread trap errno user_notif trace log

However, neither auditd nor dmesg have zero seccomp log entries printed :(

However, these don't have any effect in terms of logging with auditd in Flatcar Linux. This is why I decided to seek help from the community here.

Hopefully, someone who has experience reading Linux kernel source code and setting up seccomp logging (be it auditd or something else) will chime in and explain how this all works.

@tormath1 I appreciate your help, truly. Thank you.

[tormath1]

@igcherkaev I just came across this blogpost: https://kubernetes.io/blog/2022/12/02/seccomp-notifier/

We're simply not able to see the whole impact of that, especially because blocked syscalls via SCMP_ACT_ERRNO do not provide any additional audit logging on the system.

Maybe you should give a try to SecComp notifier?

[igcherkaev]

Thanks @tormath1 once again. So after reading that blog post, it feels like I am going to need https://kinvolk.io/blog/2022/03/bringing-seccomp-notify-to-runc-and-kubernetes ( https://github.com/kinvolk/seccompagent/blob/main/deploy/seccompagent.yaml ), which will produce the necessary logs. I'll give it a try!

[tormath1]

@igcherkaev you might be interested with https://github.com/inspektor-gadget/inspektor-gadget/blob/main/docs/builtin-gadgets/audit/seccomp.md too.