Kubernetes installable under $HOME, without the root privileges
Clone or download
AkihiroSuda Merge pull request #57 from AkihiroSuda/wip
update rootlesskit & containerd
Latest commit 9b64384 Nov 9, 2018

README.md

Usernetes: Moby (aka Docker) & Kubernetes, without the root privileges

Usernetes aims to provide a binary distribution of Moby (aka Docker) and Kubernetes that can be installed under a user's $HOME and does not require the root privileges.

Status

  • Moby (dockerd): Almost usable (except Swarm-mode)
  • Kubernetes: Early POC with a single node
    • dockershim
    • CRI-O
    • containerd

How it works

Usernetes executes Moby (aka Docker) and Kubernetes without the root privileges by using unprivileged user_namespaces(7), mount_namespaces(7), and network_namespaces(7).

To set up NAT across the host and the network namespace without the root privilege, Usernetes uses a usermode network stack (slirp4netns).

No SETUID/SETCAP binary is needed. except newuidmap(1) and newgidmap(1), which are used for setting up user_namespaces(7) with multiple sub-UIDs and sub-GIDs.

Requirements

  • newuidmap and newgidmap need to be installed on the host. These commands are provided by the uidmap package on most distros.

  • /etc/subuid and /etc/subgid should contain more than 65536 sub-IDs. e.g. penguin:231072:65536. These files are automatically configured on most distros.

$ id -u
1001
$ whoami
penguin
$ grep ^$(whoami): /etc/subuid
penguin:231072:65536
$ grep ^$(whoami): /etc/subgid
penguin:231072:65536

Distribution-specific hint

Debian (excluding Ubuntu)

  • sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone" is required

Arch Linux

  • sudo sh -c "echo 1 > /proc/sys/kernel/unprivileged_userns_clone" is required

openSUSE (and SLES(?))

  • sudo modprobe ip_tables iptable_mangle iptable_nat iptable_filter is required. (This is likely to be required on other distros as well)
  • sudo prlimit --nofile=:65536 --pid $$ is required for Kubernetes

Restrictions

Moby (dockerd):

  • Only vfs graphdriver is supported. However, on Ubuntu and a few distros, overlay2 and overlay are also supported. Starting with Linux 4.18, we will be also able to implement FUSE snapshotters.
  • Cgroups (including docker top) and AppArmor are disabled at the moment. (FIXME: we could enable Cgroups if configured on the host)
  • Checkpoint is not supported at the moment.
  • Running rootless dockerd in rootless/rootful dockerd is also possible, but not fully tested.
  • You can form Swarm-mode clusters but overlay networking is not functional.

CRI-O & containerd:

  • To be documented (almost same as Moby)

Kubernetes:

  • Multi-node networking is untested

Install from binary

Download the latest usernetes-x86_64.tbz from Releases.

$ tar xjvf usernetes-x86_64.tbz
$ cd usernetes

Install from source

$ git clone https://github.com/rootless-containers/usernetes.git
$ cd usernetes
$ go get -u github.com/go-task/task/cmd/task
$ task -d build

Quick start

Start Kubernetes using Docker

$ ./run.sh

Start Kubernetes using CRI-O

$ ./run.sh default-crio

Start Kubernetes using containerd

$ ./run.sh default-containerd

Start dockerd only (No Kubernetes)

If you don't need Kubernetes:

$ ./run.sh dockerd

Use docker

$ docker -H unix:///run/user/1001/docker.sock info

Or

$ ./dockercli.sh info

Use kubectl

$ nsenter -U -n -t $(cat $XDG_RUNTIME_DIR/usernetes/rootlesskit/child_pid) hyperkube kubectl --kubeconfig=./localhost.kubeconfig get nodes

Or

$ ./kubectl.sh get nodes

Reset to factory defaults

$ ./cleanup.sh

Advanced guide

Expose netns ports to the host

As Usernetes runs in a network namespace (with slirp4netns), you can't expose container ports to the host by just running docker run -p or kubectl expose --type=NodePort.

In addition, you need to expose Usernetes netns ports to the host via socat.

e.g.

$ pid=$(cat $XDG_RUNTIME_DIR/usernetes/rootlesskit/child_pid)
$ socat -t -- TCP-LISTEN:8080,reuseaddr,fork EXEC:"nsenter -U -n -t $pid socat -t -- STDIN TCP4\:127.0.0.1\:80"

Routing ping packets

To route ping packets, you need to set up net.ipv4.ping_group_range properly as the root.

$ sudo sh -c "echo 0   2147483647  > /proc/sys/net/ipv4/ping_group_range"

License

Usernetes is licensed under the terms of Apache License Version 2.0.

The binary releases of Usernetes contain files that are licensed under the terms of different licenses: