A tool for creating multi-node Kubernetes clusters on a Linux machine using kubeadm & systemd-nspawn. Brought to you by the Kinvolk team.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

README.md

kube-spawn Logo

kube-spawn

kube-spawn is a tool for creating a multi-node Kubernetes (>= 1.8) cluster on a single Linux machine, created mostly for developers of Kubernetes but is also a Certified Kubernetes Distribution and, therefore, perfect for running and testing deployments locally.

It attempts to mimic production setups by making use of OS containers to set up nodes.

Demo

asciicast

Requirements

  • systemd-nspawn in at least version 233

  • Large enough /var/lib/machines partition.

    If /var/lib/machines is not its own filesystem, systemd-nspawn will create /var/lib/machines.raw and loopback mount it as a btrfs filesystem. You may wish to increase the default size:

    machinectl set-limit 20G

    We recommend you create a partition of sufficient size, format it as btrfs, and mount it on /var/lib/machines, rather than letting the loopback mechanism take hold.

    In the event there is a loopback file mounted on /var/lib/machines, kube-spawn will attempt to enlarge the underlying image /var/lib/machines.raw on cluster start, but this can only succeed when the image is not in use by another cluster or machine. Not enough disk space is a common source of error. See doc/troubleshooting for instructions on how to increase the size manually.

  • qemu-img

Installation

kube-spawn should run well on a modern Linux system (for example Fedora 27 or Debian testing). If you want to test it in a controlled environment, you can use Vagrant.

To install kube-spawn on your machine, download a single binary release or build from source.

kube-spawn uses CNI to setup networking for its containers. For that, you need to download the CNI plugins (v.0.6.0 or later) from GitHub.

Example:

cd /tmp
curl -fsSL -O https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz
sudo mkdir -p /opt/cni/bin
sudo tar -C /opt/cni/bin -xvf cni-plugins-amd64-v0.6.0.tgz

By default, kube-spawn expects the plugins in /opt/cni/bin. The location can be configured with --cni-plugin-dir= from the command line or by setting cni-plugin-dir: ... in the configuration file.

Alternatively, you can use go get to fetch the plugins into your GOPATH:

go get -u github.com/containernetworking/plugins/plugins/...

Quickstart

Create and start a 3 node cluster with the name "default":

sudo ./kube-spawn create
sudo ./kube-spawn start [--nodes 3]

Reminder: if the CNI plugins can't be found in /opt/cni/bin, you need to pass --cni-plugin-dir path/to/plugins.

create prepares the cluster environment in /var/lib/kube-spawn/clusters.

start brings up the nodes and configures the cluster using kubeadm.

Shortly after, the cluster should be initialized:

[...]

Cluster "default" initialized
Export $KUBECONFIG as follows for kubectl:

        export KUBECONFIG=/var/lib/kube-spawn/clusters/default/admin.kubeconfig

After another 1-2 minutes the nodes should be ready:

export KUBECONFIG=/var/lib/kube-spawn/clusters/default/admin.kubeconfig
kubectl get nodes
NAME                          STATUS    ROLES     AGE       VERSION
kube-spawn-c1-master-q9fd4y   Ready     master    5m        v1.9.6
kube-spawn-c1-worker-dj7xou   Ready     <none>    4m        v1.9.6
kube-spawn-c1-worker-etbxnu   Ready     <none>    4m        v1.9.6

Configuration

kube-spawn can be configured by command line flags, configuration file (default /etc/kube-spawn/config.yaml or --config path/to/config.yaml), environment variables or a mix thereof.

Example:

# /etc/kube-spawn/config.yaml
cni-plugin-dir: /home/user/code/go/bin
cluster-name: cluster1
container-runtime: rkt
rktlet-binary-path: /home/user/code/go/src/github.com/kubernetes-incubator/rktlet/bin/rktlet

CNI plugins

kube-spawn supports weave, flannel, calico. It defaults to weave.

To configure with flannel:

kube-spawn create --pod-network-cidr 10.244.0.0/16 --cni-plugin flannel --kubernetes-version=v1.10.5
kube-spawn start --cni-plugin flannel --nodes 5

To configure with calico:

kube-spawn create --pod-network-cidr 192.168.0.0/16 --cni-plugin calico --kubernetes-version=v1.10.5
kube-spawn start --cni-plugin calico --nodes 5

To configure with canal:

kube-spawn create --pod-network-cidr 10.244.0.0/16 --cni-plugin canal --kubernetes-version=v1.10.5
kube-spawn start --cni-plugin canal --nodes 5

Accessing kube-spawn nodes

All nodes can be seen with machinectl list. machinectl shell can be used to access a node, for example:

sudo machinectl shell kube-spawn-c1-master-fubo3j

The password is root.

Documentation

See doc/

Building

To build kube-spawn in a Docker build container, simply run:

make

Optionally, install kube-spawn under a system directory:

sudo make install

PREFIX can be set to override the default target /usr.

Troubleshooting

See doc/troubleshooting

Community

Discuss the project on Slack.