feat: make it possible for lima to provide both

[afbjorklund]

Allow installing a container engine on your kubernetes node, and use the same lima instance for both containers and kubernetes.

For instance by installing "nerdctld" for containerd, or by running with Podman/CRI-O or Docker/CRI-Docker custom setup.

What does this PR do?

Screenshot/screencast of this PR

The same VM, running both "containers" and "kubernetes":

You configure what Docker socket filename you want to use:

What issues does this PR fix or reference?

Closes #5206

How to test this PR?

The default templates don't have any such socket (only containerd), so it would require a custom installation.

Podman: lima sudo apt install -y podman. This will install podman version 3.4.4 (or updates)

$ lima sudo podman --remote version
Client:
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.18.1
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64
Server:
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.18.1
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Docker: lima sudo apt install -y docker.io. This will install docker version 20.10 (or updates)

$ lima sudo docker version
Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~22.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default
Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.3
  Git commit:       24.0.5-0ubuntu1~22.04.1
  Built:            Mon Aug 21 19:50:14 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.7
  GitCommit:        8c087663b0233f6e6e2f4515cee61d49f14746a8
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfcb
 docker-init:
  Version:          0.19.0
  GitCommit:        

Then the unix socket for the container engine needs to be forwarded (see "podman" or "docker" templates).

The VM would run both the containers and the Kubernetes cluster, but they would not be sharing images.

So you would have to download the images (from podman/docker), and upload them again (to k3s or k8s)...

(Advanced) To share image storage:

You would install either Podman + CRI-O or Docker + CRI-Docker, and then set up both the socket and the config.

But the default container runtime is containerd, and it does not have a Docker API socket - only the nerdctl CLI.

The support for nerdctld is experimental, it can be found here: https://github.com/afbjorklund/nerdctld

You will need to install nerdctl and buildkit, in addition to the already running containerd (for k3s or k8s).

Client:
 Version:           24.0.5
 API version:       1.42 (downgraded from 1.43)
 Go version:        go1.20.3
 Git commit:        24.0.5-0ubuntu1~20.04.1
 Built:             Mon Aug 21 19:50:14 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: 🤓
 nerdctl:
  Version:          1.6.2
 containerd:
  Version:          1.7.6-k3s1.27
 Engine:
  Version:          1.6.2
  API version:      1.42 (minimum version 1.24)
  Go version:       go1.21.1
  Git commit:       e3dc23be348efded17d2cd244397b4f7018e0794
  Built:            
  OS/Arch:          linux/amd64
  Experimental:     true

You could also start with the container runtime, and add the Kubernetes installation (complete with CRI/CNI).

For podman you would need to add cri-o, and for docker you would need to add cri-dockerd. And Kubernetes.

• https://docs.k3s.io/installation

• https://kubernetes.io/docs/setup/

[afbjorklund]

Alternatively one could install cri-o on the podman machine, but I don't know how to do that.

Since Fedora CoreOS is read-only, it is more effort compared to doing it on Ubuntu or CentOS.

limactl start template://ubuntu-lts

limactl start template://centos-stream

---

Something like: (i.e. merge the templates)

copyToHost:
- guest: "/etc/kubernetes/admin.conf"
  host: "{{.Dir}}/copied-from-guest/kubeconfig.yaml"
  deleteOnStop: true
portForwards:
- guestSocket: "/var/run/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
  To run `kubectl` on the host (assumes kubectl is installed), run the following commands:
  ------
  export KUBECONFIG="{{.Dir}}/copied-from-guest/kubeconfig.yaml"
  kubectl ...
  ------
  To run `docker` on the host (assumes docker-cli is installed), run the following commands:
  ------
  export DOCKER_HOST=unix://{{.Dir}}/sock/docker.sock
  docker ...
  ------

Ready to run examples are available here:

https://github.com/lima-vm/lima/pull/1991/files

[deboer-tim]

It should probably have said "Docker" and "Kubernetes" for the connection name, instead of "Lima" and "Lima"

Could the user give these a name when they create, similar to Podman machines or Kind instances? If not, then differentiating them would be great but I think we need to keep Lima in the title - otherwise you'll see these in places like
the system tray or other UI controls (e.g. see the 'environment' in some of the Kube mockups) and without the context it could be confusing.

[afbjorklund]

They have names (the instances/VMs), but the issue here is that it is the same name for both connections...

But it is quite obvious, the only issue is that "Docker endpoint" is not shown - only "Kubernetes endpoint"
If you go to the up-right-arrow page, then it will show the details about each connection (name and url)

There is some other issue about where would be a good place to post the Lima instance name, in the status.
One alternative would be to just show "lima-k8s" like the "podman-machine-default", but then it's "too long"

So I think I will just leave it as-is, maybe revisit when lima instances work like podman machines do today.

---

$ podman-remote-static machine ls
NAME                    VM TYPE     CREATED        LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default  qemu        6 minutes ago  Currently running  1           2GiB        100GiB
$ limactl ls k8s
NAME    STATUS     SSH            VMTYPE    ARCH      CPUS    MEMORY    DISK      DIR
k8s     Stopped    127.0.0.1:0    qemu      x86_64    4       4GiB      100GiB    ~/.lima/k8s

    name: prettyMachineName(machineInfo.name),
    type: 'podman',
    status: () => podmanMachinesStatuses.get(machineInfo.name),

function prettyMachineName(machineName: string): string {
  let name;
  if (machineName === 'podman-machine-default') {
    name = 'Podman Machine';
  } else if (machineName.startsWith('podman-machine-')) {
    const sub = machineName.substring('podman-machine-'.length);
    name = `Podman Machine ${sub}`;
  } else {
    name = machineName;
  }
  return name;
}

For now, the GUI has a single text field with the name of the instance... So you need to know the name anyway.

[afbjorklund]

If you change dockerd to use containerd for storage, then the images will be shared between them:

https://docs.docker.com/storage/containerd/

Similar to how it works with podman and cri-o

Similarly, you can configure docker to use buildkitd for building, instead of the old docker builder.

[benoitf]

miss that one

[benoitf]

would be better with unit tests