Platform: Mac K8 installation is not working Docker

[sehz]

Relevant link: kubernetes/minikube#11193.

Docker does not provide way for host access container port directly.

[sehz]

This is not verified to working with Docker. This have to be done with cluster installer reworking

[sehz]

closing this for now unless someone requested support for Docker for Mac

[sehz]

Hyperkit driver works for minikube here: https://www.fluvio.io/docs/get-started/mac/. closing it

[sehz]

hyperkit doesn't work on M1 so need another workaround

[sehz]

Latest K3d removed --no-hostip - option so need solution quickly

[sehz]

Key issue is that Docker doesn't forward ports from minikube containers. This can be work around in the kind by creating using this config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  # port forward 80 on the host to 80 on this node
  extraPortMappings:
  - containerPort: 30007
    hostPort: 30007
    # optional: set the bind address on the host
    # 0.0.0.0 is the current default
    listenAddress: "127.0.0.1"
    # optional: set the protocol to one of TCP, UDP, SCTP.
    # TCP is the default
    protocol: TCP

Assuming fluvio adds node port mapping for public services

apiVersion: v1
kind: Service
metadata:
  name: fluvio-sc-public
  annotations:
    {{- toYaml .Values.loadBalancer.serviceAnnotations | nindent 4 }}
spec:
  type: {{ .Values.service.type }}
  selector:
    app: fluvio-sc
{{ if .Values.service.externalTrafficPolicy }}
    externalTrafficPolicy: .Values.service.externalTrafficPolicy
{{ end }}
{{ if .Values.service.externalName }}
    externalName: .Values.service.externalName
{{ end }}
  ports:
  - protocol: TCP
    port: 9003
    targetPort: 9003
    nodePort: 30007

The minikube command for port forwarding doesn't work on current version and it is not customizable as kind:

minikube start --driver=docker --extra-config=apiserver.service-node-port-range=32760-32767 --ports=127.0.0.1:32760-32767:32760-32767

[sehz]

All container system (including docker) is using QEMU HVF. Currently available container runtime are:

• Docker
• Podman
• Lima, VM with containers

Main issue with Docker is that it can't be used in Github Action CI due to license restriction. Given this, we should move toward standard container runtime to simplify our development and testing.

Criteria for selecting runtime are:

• License and Ease of installation
• Ease of port forwarding
• Compatibility with many Kubernetes runtime much as possible.

[sehz]

Podman doesn't work with Kind yet:

 KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster
using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Cgroup controller detection is not implemented for Podman. If you see cgroup-related errors, you might need to set systemd property "Delegate=yes", see https://kind.sigs.k8s.io/docs/user/rootless/
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 
 ✗ Preparing nodes 📦  
ERROR: failed to create cluster: podman run error: command "podman run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume 56dcb311adfa416309a7348d0fbe394ba80ce4267330b95257f6574818b264a3:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:50715:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6" failed with error: exit status 126
Command Output: Error: error configuring network namespace for container 47704ccffa0ec65232dacc224c606d900f71d08e4e51232bfc43733bab2b3f90: error adding pod kind-control-plane_kind-control-plane to CNI network "kind": failed to list chains: running [/usr/sbin/ip6tables -t nat -S --wait]: exit status 3: modprobe: ERROR: could not insert 'ip6_tables': Operation not permitted
ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.

kubernetes-sigs/kind#2537

[sehz]

Rancher desktop seems to provide comprehensive support for Kubernetes for both Mac, Windows and Linux. Currently Apple Silicon is in-progress. Rancher desktop is built on of Lima

[sehz]

For VM, Lima seems to be more flexible since it supports many distribution while Podman seems to support primary fedora. Overall, ecosystem is still unstable for making judgement although Lima seems to be most promising.