Introduction

myrootfs is a BusyBox based root file system that can be used to build very small containers for Linux.

The default x86_64 build weighs in at 3.1 MiB with a full GLIBC and BusyBox install. With a uClibc-ng based toolchain the image is no more than 1.3 MiB, in total.

Building

To try it out, clone this repository, install the dependencies listed in the Requirements section, including the toolchain for your preferred target(s). There is no default arch, the following are supported:

• arm
• arm64
• ppc
• x86_64

This example builds a SquashFS root file system for an x86_64 target from a clean checkout:

$ export PATH=/usr/local/x86_64-unknown-linux-gnu-7.3.0-1/bin:$PATH
$ ARCH=x86_64 make x86_64_defconfig
$ time make -j5
real    1m17,908s
user    2m13,820s
sys     0m37,100s

The resulting image file:

$ ls -lh images/
total 3,1M
-rw-r--r-- 1 jocke jocke 3,1M dec 28 19:27 myrootfs.img

Note: parallel builds (-j5 above) can be very hard to debug since the output of many different components is mixed. To avoid this and maintain your sanity, add --output-sync=recurse to your make -jN commands.

Requirements

The build environment currently requires at least the following tools, tested on Ubuntu 16.04 (x86_64):

$ sudo apt install build-essential libssl-dev automake autoconf libtool \
           pkg-config flex bison wget gawk quilt bc lzop libelf-dev

You also need cross-compiler toolchains. The [myRootFS][] project provides pre-built sysrooted crosstool-NG based toolchains that work on most Linux distributions that use the same, or newer, version of GLIBC as Ubuntu 16.04:

• arm-unknown-linux-gnueabi
• aarch64-unknown-linux-gnu
• powerpc-unknown-linux-gnu
• x86_64-unknown-linux-gnu

Install the toolchains in /usr/local on your build host to match the instructions below, or set your $PATH to point to the location of the bin directory.

Experimental GCC v9.2 toolchains, based on GLIBC or uClibc-ng, are also available. Change the CONFIG_TOOLCHAIN_PREFIX in your myrootfs .config, manually or using make menuconfig. For download, see our crosstool-NG releases page.

Troubleshooting

When something does not build it can be hard to see what went wrong, so there are several shortcuts and other tricks in the build system to help you. For instance, verbose mode:

$ make V=1

This is what you are probably used to from other build systems. But what if you only want to rebuild a single package?

$ make V=1 packages/busybox-build

This builds only BusyBox, with verbose mode enabled. Other useful shortcuts are:

$ make packages/busybox-clean
$ make packages/busybox-distclean
$ make packages/busybox-install

To tweak the kernel the following build shortcuts are available:

$ make kernel
$ make kernel_menuconfig
$ make kernel_saveconfig

There are a few more, see the Makefile for details.

Note: debugging Makefiles can be a bit of a hassle. To see what is really going on you can used make --debug=FLAGS V=1, or even try make SHELL='sh -x' --debug=FLAGS V=1. Consult the GNU make man page for help with the debug FLAGS.

LXD

The easiest way to test your container is to utilize myrootfs' LXD integration. To use it, you need the following packages installed on your host system:

• LXD. Pretty self-explanatory.
• Statically linked QEMU Usermode binaries. Only the statically linked variety will do, as that will be the only native binary in the root filesystem.
• Linux kernel support for binfmt_misc, with an entry mapping the foreign architecture to the static qemu binary.

NOTE: Your container does not have too be built against your host's architecture. Myrootfs automatically sets up LXD to use QEMU usermode emulation in combination with Linux binfmt_misc support, to transparently run foreign architecture binaries.

On Debian derivatives (e.g. Ubuntu), this should get you there:

$ sudo apt install lxd qemu-user-static
$ sudo adduser $LOGNAME lxd

The qemu-user-static package automatically sets up the required binfmt_misc entries, other distributions might require installing another package, or setting them up manually.

To test if everything is in order, run:

$ make lxd-prepare

And you should see something resembling this:

Checking if LXD is installed:                     /usr/bin/lxd
Checking if current user is in the lxd group:     Yes
Checking if qemu-ppc-static is available:         /usr/bin/qemu-ppc-static
Checking for ppc emulation using binfmt_misc:     Found
Installing qemu-ppc-static LXD profile:           Done
Installing myrootfs LXD profiles:                 Done

You're all set! To start your container, run:

$ make lxd-run

To exit the container, halt the system using poweroff(8) or similar, or press C-a q.

LXC config

Having built your first application image using myrootfs. Create a Linux container with LXC using:

$ sudo mkdir -p /var/lib/lxc/images/
$ sudo mkdir -p /var/lib/lxc/foo
$ sudo cp images/myrootfs.img /var/lib/lxc/images/foo.img

The LXC config file might need some tweaking, in particular if you use different path to the .img file. The host bridge you probably want to change as well. Here we have used lxcbr0:

$ sudo sh -c "cat >>/var/lib/lxc/foo/config" <<-EOF
	lxc.uts.name = foo
	lxc.tty.max = 4
	lxc.pty.max=1024
	lxc.rootfs.path = loop:/var/lib/lxc/images/foo.img
	lxc.rootfs.options = -t squashfs
	lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed
	lxc.mount.entry=run run tmpfs rw,nodev,relatime,mode=755 0 0
	lxc.mount.entry=shm dev/shm tmpfs rw,nodev,noexec,nosuid,relatime,mode=1777,create=dir 0 0
	lxc.net.0.type = veth
	lxc.net.0.flags = up
	lxc.net.0.link = lxcbr0

	#lxc.seccomp.profile = /usr/share/lxc/config/common.seccomp
	lxc.apparmor.profile = lxc-container-default-with-mounting
EOF

The last two lines are needed on systems with Seccomp and/or AppArmor. Uncomment the one you need, see the host's dmesg when lxc-start fails with mysterious error messages. For convenience the Debian/Ubuntu is uncommented already.

Note: you may have to add the following two lines to your AppArmor profile to enable writable /etc, /var, /home, and /root directories. The file is in /etc/apparmor.d/lxc/lxc-default-with-mounting:

mount fstype=tmpfs,
mount fstype=overlay,

Start your container with:

$ sudo lxc-start -n foo

To see what actually happens when it starts up, append -F. Attach to the container's /dev/console with:

$ sudo lxc-console -n foo -t 0 -e '^p'

The last -e '^p remaps the control key sequence to detach from your container and return to your host: Ctrl-p q

Bugs & Feature Requests

Feel free to report bugs and request features, or even submit your own pull requests at GitHub.

Licensing & References

With the exceptions listed below, myrootfs is distributed under the terms of the ISC License¹. myrootfs is the build system, or glue, that ties the various Open Source components together. Some files have a different license statement, e.g. kconfig. Those files are licensed under the license contained in the file itself.

myrootfs bundles patch files, which are applied to the sources of the various Open Source packages. Those patches are not covered by the license of myrootfs. Instead, they are covered by the license of the software to which the patches are applied, when said software comes available under multiple licenses, sometimes as alternative commercial licenses, the patches are provided under the publicly available Open Source licenses.

---

¹ "... functionally equivalent to the simplified BSD and MIT licenses, but without language deemed unnecessary following the Berne Convention." --Theo de Raadt