Bare metal, self-hosted, self-healing/provisioning, mesh network kubernetes cluster
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bootcfg-config
docs/blink-key
machines
manifests
networks
old
.gitignore
LICENSE
README.md
deploy
kubectl
rebuild-vm
ssh.sh
teardown
teardown-domain
teardown-net

README.md

kubermesh

Goal

  1. Deploy to one machine. Say, boot off a USB hard drive.
  2. Plug other machines into the first one. Watch them boot and join the cluster.
  3. Keep going till you run out of machines.

No need for a traditional network with switches.

Easy replacement, just drop new machines in. And the same goes for adding capacity

This repo is a prototype trying to achieve the above concepts. For more about the project, see the blog post

Local dev prerequisites

sudo apt-get install qemu-kvm libvirt-bin docker virtinst

Getting started

./deploy libvirt 4

This will set up 4 nodes using libvirt, and run the bootstrap process

Usernames/Passwords are currently hardcoded to core/core

Useful commands

virt-viewer kubermesh1 - to get a graphical console virsh console kubermesh1 - to get a serial console

Cleaning up

./teardown

Current address allocations

Anycast addresses:

  • apiserver: fd65:7b9c:569:680:98eb:c508:eb8c:1b80
  • etcd: fd65:7b9c:569:680:98eb:c508:ea6b:b0b2
  • docker hub mirror: [fd65:7b9c:569:680:98e8:1762:7b6e:83f6]:5000
  • gcr.io mirror: [fd65:7b9c:569:680:98e8:1762:7b6e:61d3]:5002
  • quay.io mirror: [fd65:7b9c:569:680:98e8:1762:7abd:e0b7]:5001

libvirt networking:

  • host <-> gateway: 172.30.0.1/30
  • host <-> gateway: 2001:db8:a001::1/64
  • gateway <-> kubermesh1: 172.30.0.9/29
  • gateway <-> kubermesh1: 2001:db8:a002::10/126
  • gateway docker: 172.30.1.1/24
  • flannel ipv4: 172.31.0.0/16
  • custom ipv4 allocation: 172.30.2.0/24
  • custom ipv6 allocation: 2001:db8::/71

Installation networking:

  • ??::0/119
  • host: ::0/123
  • host vip: ::0/128
  • mesh interfaces: ::10/123
  • mesh interface 1: ::10/126
  • mesh interface 2: ::14/126
  • mesh interface 3: ::18/126
  • mesh interface 4: ::1c/126
  • pods: ::100/120

Hardware specific

NUCs

  • Disable legacy network boot in the bios, so it will use EFI boot over ipv6
    • Enter the BIOS
    • Select Advanced under Boot Order
    • Under Legacy Boot Priority disable Legacy Boot
    • Under Boot Configuration select Boot Network Devices Last
    • Under Boot Configuration select Unlimited Boot to Network Attempts
    • Under Boot Configuration check Network Boot is set to UEFI PXE & iSCSI
    • F10

USB Devices for the gateway bootstrapping physical hardware

  • Plug USB3 ones into a USB2 port, so qemu can use it without Speed Mismatch errors
  • Might need to add /run/udev/data/** r, to /etc/apparmor.d/abstractions/libvirt-qemu until qemu fixes that bug