Add usernetes with docker example

[afbjorklund]

Will run Kubernetes in rootless docker, a.k.a. "usernetes".

• https://github.com/rootless-containers/usernetes

• https://rootlesscontaine.rs/getting-started/docker

Add certificate for localhost so we can use it from the host.

Note: This is running kubernetes-in-docker (kind)

Currently: FROM docker.io/kindest/node:v1.28.0

NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION      CONTAINER-RUNTIME
u7s-lima-u7s   Ready    control-plane   15m   v1.28.0   192.168.5.15   <none>        Debian GNU/Linux 11 (bullseye)   5.15.0-86-generic   containerd://1.7.1

---

CONTAINER ID   IMAGE            COMMAND                  CREATED          STATUS          PORTS                                                                                                                                                                            NAMES
a666a9b648cb   usernetes-node   "/u7s-entrypoint.sh …"   20 minutes ago   Up 15 minutes   0.0.0.0:2379->2379/tcp, :::2379->2379/tcp, 0.0.0.0:6443->6443/tcp, :::6443->6443/tcp, 0.0.0.0:10250->10250/tcp, :::10250->10250/tcp, 0.0.0.0:8472->8472/udp, :::8472->8472/udp   usernetes-node-1

REPOSITORY       TAG                      IMAGE ID       CREATED          SIZE
usernetes-node   latest                   f5a3eccc00a4   22 minutes ago   1.09GB
kindest/node     v1.28.0                  ad70201dab13   3 months ago     950MB
debian           bullseye-20230814-slim   fb4ec5ceea2f   3 months ago     80.5MB
busybox          latest                   a416a98b71e2   4 months ago     4.26MB

[afbjorklund]

It is possible to run this example with podman, by changing some variables.

export CONTAINER_ENGINE=podman

[AkihiroSuda]

Can we us yq?

[afbjorklund]

Need to install it.

$ yq
Command 'yq' not found, but can be installed with:
snap install yq
Please ask your administrator.

[vsoch]

@afbjorklund how are you handling creating the join-command here (and sharing with a worker?) I'm trying to get that working for my setup, and it seems like the mounts are not done until the provision is done (so I can't write content there) and there doesn't seem to be a copy directive for into the build. I found limactl copy but that assumes both are running (and I'd like the second to provision with the file) https://fig.io/manual/limactl and I found this issue #594 but it wasn't clear that there was a resolution. Thanks for the tips!

[jandubois]

it seems like the mounts are not done until the provision is done

This is true for reverse-sshfs, but not for 9p or virtiofs.

[vsoch]

This is true for reverse-sshfs, but not for 9p or virtiofs.

Is there an example that shows how to do this? And so I understand, the reverse sshfs is like a network filesystem (using ssh) and then the others are different kinds of filesystems? I found:

• https://virtio-fs.gitlab.io/
• is this the right 9p? https://en.wikipedia.org/wiki/9P_(protocol) it's a hard one to search for! 😆

[jandubois]

lima/examples/default.yaml

Lines 101 to 104 in a21b5f3

	# Mount type for above mounts, such as "reverse-sshfs" (from sshocker), "9p" (EXPERIMENTAL, from QEMU’s virtio-9p-pci, aka virtfs),
	# or "virtiofs" (EXPERIMENTAL, needs `vmType: vz`)
	# 🟢 Builtin default: "reverse-sshfs" (for QEMU), "virtiofs" (for vz)
	mountType: null

[afbjorklund]

how are you handling creating the join-command here (and sharing with a worker?)

I did not, it was something that was left-do-do also for the regular rootful Kubernetes...

• Multi-node kubernetes clusters #1184

I think it will most likely involve a lima wrapper, with appropriate "hooks" in the scripts?

#1184 (comment)

---

But this is using docker compose, so that would be an alternative (e.g. using a volume)

For regular kind, you would just ask for another node in the config file (in the yaml)...

[afbjorklund]

• is this the right 9p?

https://wiki.qemu.org/Documentation/9psetup

You can also search for "VirtFS", the qemu setup:
VirtFS—A virtualization aware File System pass-through [PDF]

But yeah, it is the same old Plan 9 transport...

[afbjorklund]

Note: we scoped out the regular kind and k3d templates, and left it as an exercise for the reader

• Add kubernetes with kind example #596

So we could do the same with usernetes, and just refer to "docker.yaml" (rootless) as the base?

Especially if the script grows any longer (like k8s)

[vsoch]

For those following the thread about virtio-fs - I found that the C variant that shipped with QEMU did not work. So I built the rust one, e.g.,

# This is in the PWD
git clone https://gitlab.com/virtio-fs/virtiofsd 
cd virtiofsd 
sudo apt install libcap-ng-dev libseccomp-dev

Then build with cargo.

cargo build --release

Then I replaced it.

sudo mv /usr/lib/qemu/virtiofsd /usr/lib/qemu/virtiofsd-c
sudo mv virtiofsd/target/release/virtiofsd /usr/lib/qemu/virtiofsd

I also did:

sudo usermod -aG kvm $USER

And then I was able to copy the join-command (or whatever files I needed) into the mount directory and they will be present when you are provisioning!

mkdir -p /tmp/lima
cp /home/vanessa/.lima/control-plane/join-command /tmp/lima/join-command

@afbjorklund @jandubois I removed my home ~ for this to work, but then it shells in and there is no such file or directory. Is there a way to change the present working directory? I already checked default.yaml and searched around for terms but didn't find anything. ty!

[jandubois]

I removed my home ~ for this to work, but then it shells in and there is no such file or directory. Is there a way to change the present working directory?

I'm not sure what you mean by "it shells in". In general you can specify the working directory with limactl shell $INSTANCE --workdir $WORKDIR ... and for the lima wrapper you can set the default with $LIMA_WORKDIR.

But I'm not sure if this applies to your situation as I'm not clear about what "it" means...

[vsoch]

This works great!

limactl shell --workdir /opt/usernetes usernetes-worker

that should work for now. Thank you!

[vsoch]

Thanks for the help! I now have a worker and control plane set of configs, and it's entirely automated except for copying the join-command to the (TBA worker) directory that is mounted and available for provision, and then for actually running join I have the user do that interactively because otherwise there is an issue with containerd. But overall it's just a few commands and very easy, and that's great! Thanks for all the help! https://github.com/converged-computing/usernetes-lima

[afbjorklund]

Added a very important sleep 5 (for containerd to start), and moved make kubeconfig above kubectl (doh!)

It would be nice if usernetes used something like /etc/usernetes for the kubeconfig, but that's another issue.

EDIT: well, something rootless (not /etc)

Guess it will just be something in home.

[AkihiroSuda]

Thanks

[afbjorklund]

To access the control-plane (which is normally not needed, but whatever) one can do:

limactl shell --workdir usernetes u7s make shell

And like it says in the README, you need to edit the compose.yaml in order to add node ports.

But when you do, they are also forwarded by Lima...

[AkihiroSuda]

Thanks