If you are using a released version of Kubernetes, you should refer to the docs that go with that version.
Documentation for other releases can be found at releases.k8s.io.
AppArmor is a mandatory access control (MAC) system for Linux that supplements the standard Linux user and group based permissions. AppArmor can be configured for any application to reduce the potential attack surface and provide greater defense in depth. It is configured through profiles tuned to whitelist the access needed by a specific program or container, such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either enforcing mode, which blocks access to disallowed resources, or complain mode, which only reports violations.
AppArmor is similar to SELinux. Both are MAC systems implemented as a Linux security module (LSM), and are mutually exclusive. SELinux offers a lot of power and very fine-grained controls, but is generally considered very difficult to understand and maintain. AppArmor sacrifices some of that flexibility in favor of ease of use. Seccomp-bpf is another Linux kernel security feature for limiting attack surface, and can (and should!) be used alongside AppArmor.
AppArmor can enable users to run a more secure deployment, and / or provide better auditing and monitoring of their systems. Although it is not the only solution, we should enable AppArmor for users that want a simpler alternative to SELinux, or are already maintaining a set of AppArmor profiles. We have heard from multiple Kubernetes users already that AppArmor support is important to them. The seccomp proposal details several use cases that also apply to AppArmor.
Much of this design is drawn from the work already done to support seccomp profiles in Kubernetes, which is outlined in the seccomp design doc. The designs should be kept close to apply lessons learned, and reduce cognitive and maintenance overhead.
Docker has supported AppArmor profiles since version 1.3, and maintains a default profile which is applied to all containers on supported systems.
AppArmor was upstreamed into the Linux kernel in version 2.6.36. It is currently maintained by Canonical, is shipped by default on all Ubuntu and openSUSE systems, and is supported on several other distributions.
This document describes the proposed design for alpha-level support, although post-alpha features are described in future work. For AppArmor alpha support (targeted for Kubernetes 1.4) we will enable:
- Specifying a pre-loaded profile to apply to a pod container
- Restricting pod containers to a set of profiles (admin use case)
We will also provide a reference implementation of a pod for loading profiles on nodes, but an official supported mechanism for deploying profiles is out of scope for alpha.
An AppArmor profile can be specified for either a pod or container through the Kubernetes API with a pod annotation. If a profile is specified, the Kubelet will verify that the node meets the required prerequisites (e.g. the profile is already configured on the node) before starting the container, and will not run the container if the profile cannot be applied. If the requirements are met, the container runtime will configure the appropriate options to apply the profile. Profile requirements and defaults can be specified on the PodSecurityPolicy.
When an AppArmor profile is specified, the Kubelet will verify the prerequisites for applying the profile to the container. In order to fail securely, a container will not be run if any of the prerequisites are not met. The prerequisites are:
- Kernel support - The AppArmor kernel module is loaded. Can be checked by libcontainer.
- Runtime support - For alpha, Docker will be required (rkt does not currently have AppArmor support). All supported Docker versions include AppArmor support. See Container Runtime Interface for other runtimes.
- Installed profile - The target profile must be loaded prior to starting the container. Loaded profiles can be found in the AppArmor securityfs [1].
If any of the prerequisites are not met an event will be generated to report the error and the pod will be rejected by the Kubelet.
[1] The securityfs can be found in /proc/mounts, and defaults to /sys/kernel/security on my
Ubuntu system. The profiles can be found at {securityfs}/apparmor/profiles
(example).
The intial alpha support of AppArmor will follow the pattern used by seccomp and specify profiles through annotations. Profiles can be specified through pod annotations, as either a container profile, a pod profile (applied to all pod containers), or both (in which case the container annotation overrides the pod annotation). The annotation format is a URI key, and a profile name value:
apparmor.security.alpha.kubernetes.io/container/<container_name>apparmor.security.alpha.kubernetes.io/pod
The profiles can be specified in the following formats (following the convention used by seccomp):
runtime/default- Applies the default profile for the runtime. For docker, the profile is generated from a template here. If no AppArmor annotations are provided, this profile is enabled by default if AppArmor is enabled in the kernel. Runtimes may define this to be unconfined, as Docker does for privileged pods.localhost/<profile_name>- The profile name specifies the profile to load.
Note: There is no way to explicitly specify an "unconfined" profile, since it is discouraged. If this is truly needed, the user can load an "allow-all" profile.
The PodSecurityPolicy allows cluster administrators to control the security context for a pod its containers. An annotation can be specified on the PodSecurityPolicy to restrict which AppArmor profiles can be used, and specify a default if no profile is specified.
The annotation key is apparmor.security.alpha.kubernetes.io/allowedProfileNames. The value is a
comma delimited list, with each item following the format described above. If a list
of profiles are provided and a pod does not have an AppArmor annotation, the first profile in the
list will be used by default.
Enforcement of the policy is standard. See the seccomp implementation as an example.
We will provide a reference implementation of a pod for loading profiles on nodes, but there will
not be an official mechanism or API in the alpha version (see future work).
The reference container will contain the apparmor_parser tool and a script for using the tool to
load all profiles in a set of (configurable) directories. The initial implementation will be
designed to run once to completion, as opposed to watching the directories for changes. It can be
run in a DaemonSet to load the profiles onto all nodes. The pod will need to be run in privileged
mode.
This simple design should be sufficient to deploy AppArmor profiles from any volume source, such as a ConfigMap or PersistentDisk. Users seeking more advanced features should be able extend this design easily.
Our e2e testing framework does not currently run nodes with AppArmor enabled, but we can run a node e2e test suite on an AppArmor enabled node. The cases we should test are:
- PodSecurityPolicy - These tests can be run on a cluster even if AppArmor is not enabled on the
nodes.
- No AppArmor policy allows pods with arbitrary profiles
- With a policy a default is selected
- With a policy arbitrary profiles are prevented
- With a policy allowed profiles are allowed
- Node AppArmor enforcement - These tests need to run on AppArmor enabled nodes, in the node e2e
suite.
- A valid container profile gets applied
- A valid pod profile is applied to all containers
- A valid pod and container profile uses the container profile
- An unloaded profile will be rejected
Post-alpha feature ideas. These are not fully-fleshed designs.
We should publish (to GitHub) AppArmor profiles for all Kubernetes system components, including core
components like the API server and controller manager, as well as addons like influxDB and
Grafana. kube-up.sh and its successor should have an option to apply the profiles, if the AppArmor
is supported by the nodes. Distros that support AppArmor and provide a Kubernetes package should
include the profiles out of the box.
We could provide an official supported solution for loading profiles on the nodes. One option is to extend the reference implementation described above into a DaemonSet that watches the directory sources to sync changes, or to watch a ConfigMap object directly. Another option is to add an official API for this purpose, and load the profiles on-demand in the Kubelet.
Profile stacking is an AppArmor feature currently in development that will enable multiple profiles to be applied to the same object. If profiles are stacked, the allowed set of operations is the "intersection" of both profiles (i.e. stacked profiles are never more permissive). Taking advantage of this feature, the cluster administrator could restrict the allowed profiles on a PodSecurityPolicy to a few broad profiles, and then individual apps could apply more app specific profiles on top.
AppArmor, SELinux, TOMOYO, grsecurity, SMACK, etc. are all Linux MAC implementations with similar requirements and features. At the very least, the AppArmor implementation should be factored in a way that makes it easy to add alternative systems. A more advanced approach would be to extract a set of interfaces for plugins implementing the alternatives. An even higher level approach would be to define a common API or profile interface for all of them. Work towards this last option is already underway for Docker, called Docker Security Profiles.
Other container runtimes will likely add AppArmor support eventually, so the
Container Runtime Interface (CRI) needs to be made compatible
with this design. The two important pieces are a way to report whether AppArmor is supported by the
runtime, and a way to specify the profile to load (likely through the LinuxContainerConfig).
Whether AppArmor is running in enforcing or complain mode it generates logs of policy violations. These logs can be important cues for intrusion detection, or at the very least a bug in the profile. Violations should almost always generate alerts in production systems. We should provide reference documentation for setting up alerts.
A common method for writing AppArmor profiles is to start with a restrictive profile in complain
mode, and then use the aa-logprof tool to build a profile from the logs. We should provide
documentation for following this process in a Kubernetes environment.
-
Load an AppArmor profile with
apparmor_parser(required by Docker so it should be available):$ apparmor_parser --replace --write-cache /path/to/profile -
Unload with:
$ apparmor_parser --remove /path/to/profile