The Firecracker Jailer
The jailer is invoked in this manner:
jailer --id <id> \ --node <numa_node>\ --exec-file <exec_file> \ --uid <uid> \ --gid <gid> [--chroot-base-dir <chroot_base>] [--netns <netns>] [--daemonize] [--seccomp-level <level>]
idis the unique VM identification string, which may contain alphanumeric characters and hyphens. The maximum
idlength is currently 64 characters.
numa_noderepresents the NUMA node the process gets assigned to. More details are available below.
exec_fileis the path to the Firecracker binary that will be exec-ed by the jailer. The user can provide a path to any binary, but the interaction with the jailer is mostly Firecracker specific.
gidare the uid and gid the jailer switches to as it execs the target binary.
chroot_baserepresents the base folder where chroot jails are built. The default is
netnsrepresents the path to a network namespace handle. If present, the jailer will use this to join the associated network namespace.
- When present, the
--daemonizeflag causes the jailer to cal
setsid()and redirect all three standard I/O file descriptors to
--seccomp-levelspecifies whether seccomp filters should be installed and how restrictive they should be. Possible values are:
- 0 (default): disabled.
- 1 : basic filtering. This prohibits syscalls not whitelisted by Firecracker.
- 2 : advanced filtering. This adds further checks on some of the parameters of the allowed syscalls.
After starting, the Jailer goes through the following operations:
- Validate all provided paths and the VM
- Close all open file descriptors based on
sysconf(_SC_OPEN_MAX)except input, output and error.
/dev/kvmas RW, and bind a Unix domain socket listener to
exec_file_nameis the last path component of
exec_file(for example, that would be
/usr/bin/firecracker). Both descriptors remain open across exec-ing into the target binary, which would be otherwise unable to open/create the associated files.
- Create the
<chroot_base>/<exec_file_name>/<id>/rootfolder, which will be henceforth referred to as
chroot_dir. Nothing is done if the path already exists (it should not, since
idis supposed to be unique).
- Create the
cgroupsub-folders. At the moment, the jailer uses three
pids. On most systems, these (along with others) are mounted by default somewhere in
/sys/fs/cgroup(they should be mounted by the user otherwise). The jailer will parse
/proc/mountsto detect where each of the three controllers can be found (multiple controllers may share the same path). For each identified location (referred to as
<cgroup_base>), the jailer creates the
<cgroup_base>/<exec_file_name>/<id>subfolder, and writes the current pid to
<cgroup_base>/<exec_file_name>/<id>/tasks. Also, the value of
numa_nodeis written to the appropriate
unshare()into a new mount namespace, use
pivot_root()to switch the old system root mount point with a new one base in
chroot_dir, switch the current working directory to the new root, unmount the old root mount point, and call
chrootinto the current directory.
mknodto create a
/dev/net/tunequivalent inside the jail.
--netns <netns>is present, attempt to join the specified network namespace.
--daemonizeis specified, call
- Drop privileges via setting the provided
- Exec into
context_jsonis a structure to pass parameters from the jailer to Firecracker. The json string includes the following keys:
string) - The
idargument provided to the jailer.
boolean) value (
truein this case) to let Firecracker know that it's running inside a jail.
--seccomp-levelargument provided to the jailer.
Example Run and Notes
Let’s assume Firecracker is available as
/usr/bin/firecracker, and the jailer
can be found at
/usr/bin/jailer. We pick the unique id
551e7604-e35c-42b3-b825-416853441234, and we choose to run on NUMA node
0, using uid 123, and gid 100. For this example, we are content with
/srv/jailer chroot base dir.
We start by running:
/usr/bin/jailer --id 551e7604-e35c-42b3-b825-416853441234 --node 0 --exec-file \ /usr/bin/firecracker --uid 123 --gid 100 --netns /var/run/netns/my_netns \ --daemonize
After opening the file descriptors mentioned in the previous section, the jailer will create the following resources (and all their prerequisites, such as the path which contains them):
We are going to refer to
Let’s also assume the cpu, cpuset, and pids cgroups are mounted at
respectively. The jailer will create the following subfolders (which will
inherit settings from the parent cgroup):
It’s worth noting that, whenever a folder already exists, nothing will be done,
and we move on to the next directory that needs to be created. This should only
happen for the common
firecracker subfolder (but, as for creating the chroot
path before, we do not issue an error if folders directly associated with the
id already exist).
The jailer then writes the current pid to
It also writes
--netns parameter is specified in our example, the jailer opens
/var/run/netns/my_netns to get a file descriptor
setns(fd, CLONE_NEWNET) to join the associated network namespace, and then
--daemonize flag is also present, so the jailers opens
RW and keeps the associate file descriptor as
dev_null_fd (we do this
before going inside the jail), to be used later.
Build the chroot jail. First, the jailer uses
unshare() to enter a new mount
namespace, and changes the propagation of all mount points in the new namespace
to private using
mount(NULL, “/”, NULL, MS_PRIVATE | MS_REC, NULL), as a
pivot_root(). Another required operation is to bind mount
<chroot_dir> on top of itself using
mount(<chroot_dir>, <chroot_dir>, NULL, MS_BIND, NULL). At this point, the jailer creates the folder
<chroot_dir>/old_root, changes the current directory to
syscall(SYS_pivot_root, “.”, “old_root”). The final steps of
building the jail are unmounting
umount2(“old_root”, MNT_DETACH), deleting
rmdir, and finally calling
chroot(“.”) for good measure. From now, the process is jailed in
Create the special file
mknod(“/dev/net/tun”, S_IFCHR | S_IRUSR | S_IWUSR, makedev(10, 200)), and then call
chown(“/dev/net/tun”, 123, 100), so Firecracker can use it after dropping privileges. This is
required to use multiple TAP interfaces when running jailed.
--daemonize flag is present, call
setsid() to join a new
session, a new process group, and to detach from the controlling terminal.
Then, redirect standard file descriptors to
/dev/null by calling
dup2(dev_null_fd, STDOUT), and
dup2(dev_null_fd, STDERR). Close
dev_null_fd, because it is no longer necessary.
Finally, the jailer switches the
100, and execs
We can now use the socket at
to interact with the VM.
- The user must create hard links for (or copy) any resources which will be provided to the VM via the API (disk images, kernel images, named pipes, etc) inside the jailed root folder. Also, permissions must be properly managed for these resources; for example the user which Firecracker runs as must have both read and write permissions to the backing file for a RW block device.
- It’s up to the user to load balance VM placement among multiple NUMA nodes
(if present), using the
--nodecommand line argument.
- The user must also manage any further fine tuning of resource partitioning via cgroups (most likely the ones created by the jailer), or any other means.
- It’s up to the user to handle cleanup after running the jailer. One way to do
this involves registering handlers with the cgroup
notify_on_releasemechanism, while being wary about potential race conditions (the instance crashing before the subscription process is complete, for example).
- For extra resilience, the jailer expects to be spawned by the user in a new
PID namespace, most likely via a combination of
exec(). A process must be created in a new PID namespace in order to become a pseudo-init process, and the other option is to use a
clone()in the jailer, which seems unnecessary.
- When running with
--daemonize, the jailer will fail to start if it's a process group leader, because
setsid()returns an error in this case. Spawning the jailer via
exec()also ensures it cannot be a process group leader.
- We run the jailer as the
rootuser; it actually requires a more restricted set of capabilities, but that's to be determined as features stabilize.
- The jailer can only log messages to stdout/err for now, which is why the
logic associated with
--daemonizeruns towards the end, instead of the very beginning. We are working on adding better logging capabilities.
- If all the cgroup controllers are bunched up on a single mount point using the "all" option, our current program logic will complain it cannot detect individual controller mount points.