runtime,agent: Add AppArmor support on the guest side

[ManaSugi]

Kata containers support AppArmor for containers running inside the guest in order to improve security.

Containers inside the guest are launced with the Kata default profile (kata-default) applied if disable_guest_apparmor is set to false in the runtime configuration. The kata-default based on the containerd's default profile is applied to the container process inside the guest by default. Users can also set a custom AppArmor profile to the container process using guest_apparmor_profile in the runtime configuration. This will be an alternative configuration of the Kubernetes' annotation, container.apparmor.security.beta.kubernetes.io, for AppArmor because Kata users cannot apply AppArmor profiles to the guest container via Kubernetes' configuration. To apply the profile to the container, the guest rootfs must be Ubuntu or Debian that is created with APPARMOR=yes. The kata-default is stored under /etc/apparmor.d in the rootfs. This works only if the init service is systemd, not the agent init.

Fixes: #7586

Signed-off-by: Manabu Sugimoto Manabu.Sugimoto@sony.com

[ManaSugi]

Experiments with containerd

Requirements

• Enable the guest AppArmor in the runtime configuration

sudo sed -i '/^disable_guest_apparmor/ s/true/false/g' /etc/kata-containers/configuration.toml

Or enable it using the pod-level annotation in your sandbox.json

"annotations": {
    "io.katacontainers.config.hypervisor.disable_guest_apparmor": "false"
 }

• Create and build a rootfs for AppArmor

The apparmor package is installed and the default profile (kata-default) is stored in the guest rootfs
if the APPARMOR is set to yes. The kata-default profile is copied from tools/osbuilder/rootfs-builder/template/apparmor/kata-default to /etc/apparmor.d in the guest rootfs.

$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true APPARMOR=yes ./rootfs.sh ubuntu'
$ script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh ${ROOTFS_DIR}'

Qemu or CloudHypervisor

Kata containers with qemu or cloudhypervisor via crictl can be launched.

1. Launch Kata containers with AppArmor (default profile kata-default)

The default profile kata-default is loaded and applied to container processes by default.

Launch a container using crictl

container.json

{
    "metadata": {
        "name": "kata-ubuntu"
    },
    "image": {
        "image": "docker.io/ubuntu"
    },
    "command": [
        "/bin/sh"
    ],
    "envs": [
        {
            "key": "PATH",
            "value": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        },
        {
            "key": "TERM",
            "value": "xterm"
        }
    ],
    "stdin": true,
    "stdin_once": true,
    "tty": true,
    "linux": {
    }
}

sandbox.json

{
    "metadata": {
        "name": "kata-sandbox",
        "namespace": "default",
        "uid": "kata-sandbox"
    },
    "linux": {
    }
}

$ uname -a
Linux *** 5.10.0-1057-oem #61-Ubuntu SMP Thu Jan 13 15:06:11 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ sudo crictl run -r kata container.json sandbox.json
d85f905ed2aebe5ea60dcdea65d06473b42f384ee8f018b20aa5f5d4d878be83

Attach the Kata VM

You can confirm that the kata-default profile is loaded and applied to /pause and /usr/bin/dash processes.

$ sudo kata-runtime exec e0c15b199093de09720eb0f7c4954ee27ba8737039bb92620e1f74e8a255a7e9

root@localhost:/# uname -a
Linux localhost 6.1.38 #1 SMP Fri Aug  4 10:19:09 JST 2023 x86_64 x86_64 x86_64 GNU/Linux

root@localhost:/# cat /sys/module/apparmor/parameters/enabled
Y

root@localhost:/# apparmor_status
apparmor module is loaded.
5 profiles are loaded.
5 profiles are in enforce mode.
   /usr/sbin/chronyd
   kata-default
   lsb_release
   nvidia_modprobe
   nvidia_modprobe//kmod
0 profiles are in complain mode.
4 processes have profiles defined.
4 processes are in enforce mode.
   /usr/sbin/chronyd (121)
   /usr/sbin/chronyd (122)
   /usr/bin/dash (128) kata-default
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
root@localhost:/#

2. Launch Kata containers with AppArmor (custom profile)

When you want to apply a custom AppArmor profile rather than the kata-default, you can use the guest_apparmor_profile in the runtime configuration.
For this experiment, we can try to apply the nvidia_modprobe as a custom profile, which is provided
the apparmor package and stored under /etc/apparmor.d in the guest rootfs by default.

Note:

• The nvidia_modprobe is not a suitable profile for container processes.
  Hence, this experiment is for verifying the guest_apparmor_profile works properly.
• If you want to apply your custom profile, you need to put the profile under /etc/apparmor.d in the
  guest rootfs in advance.

Specify nvida_modprobe to the guest_apparmor_profile.

$ sudo sed -i -e 's/^# *\(guest_apparmor_profile\).*=.*$/\1="nvidia_modprobe"/g' /etc/kata-containers/configuration.toml

Or use the pod-level annotation in your sandbox.json.

"annotations": {
    "io.katacontainers.config.runtime.guest_apparmor_profile": "nvidia_modprobe"
}

Launch a container using crictl

$ sudo crictl run -r kata container.json sandbox.json
d85f905ed2aebe5ea60dcdea65d06473b42f384ee8f018b20aa5f5d4d878be83

Attach the Kata VM

You can confirm that the nvidia_modprobe profile is loaded and applied to /pause and /usr/bin/dash processes
instead of the kata-default.

$ sudo kata-runtime exec 2aeca734667dc9b1bd24f544dbcf2701dfbdd8ae885cd44cf6ea16ce51a7f0ca

root@localhost:/# cat /sys/module/apparmor/parameters/enabled
Y

root@localhost:/# apparmor_status
apparmor module is loaded.
5 profiles are loaded.
5 profiles are in enforce mode.
   /usr/sbin/chronyd
   kata-default
   lsb_release
   nvidia_modprobe
   nvidia_modprobe//kmod
0 profiles are in complain mode.
4 processes have profiles defined.
4 processes are in enforce mode.
   /usr/sbin/chronyd (121)
   /usr/sbin/chronyd (122)
   /usr/bin/dash (128) nvidia_modprobe
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

Firecracker

I couldn't launch Kata Containers with firecracker via crictl (CRI), so I used ctr instead.

Launch Kata containers with AppArmor (default profile kata-default)

Launch a container using ctr

$ sudo ctr run --snapshotter devmapper --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest kata-test sh

Attach the Kata VM

You can confirm that the kata-default profile is loaded and applied to /bin/dash process.

$ sudo kata-runtime exec kata-test
root@localhost:/# apparmor_status
apparmor module is loaded.
5 profiles are loaded.
5 profiles are in enforce mode.
   /usr/sbin/chronyd
   kata-default
   lsb_release
   nvidia_modprobe
   nvidia_modprobe//kmod
0 profiles are in complain mode.
3 processes have profiles defined.
3 processes are in enforce mode.
   /usr/sbin/chronyd (129)
   /usr/sbin/chronyd (131)
   /bin/dash (139) kata-default
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

[ManaSugi]

/cc @fidencio @amshinde @liubin

[ManaSugi]

/test

[ManaSugi]

/test

[ManaSugi]

/test

[ManaSugi]

/test-power

[ManaSugi]

/test-dragonball

[ManaSugi]

/test-power

[ManaSugi]

/test

[ManaSugi]

/test

[ManaSugi]

/test-dragonball

[ManaSugi]

/test-dragonball

[ManaSugi]

Updates:
The previous implementation had applied AppArmor profiles to the pause container, but this isn’t right behavior. So if the ContainerType is PodSandbox, applying the profile is skipped.

Besides, the test kata-containers/tests#5749 passed. This PR is ready for the merge.

[ManaSugi]

@fidencio @amshinde @bergwolf @jiangliu
I'd appreciate if you could review this when you have time.

After this is merged into the mainline, I'll prepare for adding the disable_guest_apparmor option to the remote hypervisor like confidential-containers/cloud-api-adaptor#439.

[fidencio]

I'd appreciate if you could review this when you have time.

I'm adding thsi to my list, but it won't happen during this week, sorry.
Next week I will get to this one.

[github-actions]

This PR has been opened without with no activity for 180 days. Comment on the issue otherwise it will be closed in 7 days