[BUG] v4.1.0 REGRESSION --k3s-server-arg doesn't work

[mjgallag]

k3d cluster create --k3s-server-arg --disable=traefik worked in v4.0.0 but doesn't work in v4.1.0, traefik is enabled. I suspect it isn't passing any --k3s-server-args. I tried different formatting, i.e. single quotes, no equals etc ... to fix it but had no luck. This exact command worked immediately proceeding my upgrade from v4.0.0 to v4.1.0. Perhaps a regression test is in order.

Thanks, great tool!

[iwilltry42]

Hi @mjgallag , thanks for opening this issue!
This is caused by an incompatibility with flag types in Viper.
Fix is pushed and release process kicked off 👍

[mjgallag]

thanks for the quick fix!

[RainFlying]

@iwilltry42

Tried version 4.1.0, didn't fix the issue.

% k3d version
k3d version v4.1.0
k3s version v1.20.2-k3s1 (default)

With command

k3d cluster create --servers 1 --agents 3 --k3s-server-arg --disable=servicelb --k3s-server-arg --disable=traefik

 % kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-598b589c46-jxtjr   1/1     Running   0          7m37s
svclb-nginx-pfhn4        0/1     Pending   0          2m5s
svclb-nginx-62xpw        0/1     Pending   0          2m5s
svclb-nginx-zbl7d        0/1     Pending   0          2m5s
svclb-nginx-qchwl        0/1     Pending   0          2m5s

[iwilltry42]

@RainFlying , can you please give some more details?

I copy-pasted your command line and it works perfectly for me:

kubectl get all --all-namespaces                                                                                                                                                                        
Alias tip: kgaa
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   pod/local-path-provisioner-7c458769fb-jcgkm   1/1     Running   0          75s
kube-system   pod/metrics-server-86cbb8457f-rg9bd           1/1     Running   0          75s
kube-system   pod/coredns-854c77959c-xdh6x                  1/1     Running   0          75s

NAMESPACE     NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes       ClusterIP   10.43.0.1       <none>        443/TCP                  92s
kube-system   service/kube-dns         ClusterIP   10.43.0.10      <none>        53/UDP,53/TCP,9153/TCP   89s
kube-system   service/metrics-server   ClusterIP   10.43.151.193   <none>        443/TCP                  89s

NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/local-path-provisioner   1/1     1            1           89s
kube-system   deployment.apps/metrics-server           1/1     1            1           89s
kube-system   deployment.apps/coredns                  1/1     1            1           89s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/local-path-provisioner-7c458769fb   1         1         1       75s
kube-system   replicaset.apps/metrics-server-86cbb8457f           1         1         1       75s
kube-system   replicaset.apps/coredns-854c77959c                  1         1         1       75s

Maybe you can also append the output of docker inspect k3d-k3s-default-server-0 here?

[RainFlying]

docker inspect k3d-k3s-default-server-0

Hi @iwilltry42 I deleted and recreated the cluster with --verbose option.

% k3d version
k3d version v4.1.0
k3s version v1.20.2-k3s1 (default)

% k3d cluster create --servers 1 --agents 3 --k3s-server-arg --disable=servicelb --k3s-server-arg --disable=traefik --verbose

EBU[0000] Selected runtime is 'docker.Docker'
DEBU[0000] Configuration:
agents: 3
image: docker.io/rancher/k3s:v1.20.2-k3s1
network: ""
options:
  k3d:
    disablehostipinjection: false
    disableimagevolume: false
    disableloadbalancer: false
    disablerollback: false
    timeout: 0s
    wait: true
  k3s:
    extraagentargs: '[]'
    extraserverargs: '[--disable=servicelb,--disable=traefik]'
  kubeconfig:
    switchcurrentcontext: true                                                                                                                                                                           updatedefaultkubeconfig: true                                                                                                                                                                      runtime:
    gpurequest: ""
registries:
  config: ""
  create: false
  use: []
servers: 1
token: ""
DEBU[0000] Additional CLI Configuration:
cli:
  api-port: ""
  env: []
  labels: []
  ports: []
  volumes: []
DEBU[0000] ========== Simple Config ==========
{TypeMeta:{Kind: APIVersion:} Name: Servers:1 Agents:3 ExposeAPI:{Host: HostIP: HostPort:} Image:docker.io/rancher/k3s:v1.20.2-k3s1 Network: ClusterToken: Volumes:[] Ports:[] Labels:[] Options:{K3d
Options:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false PrepDisableHostIPInjection:false NodeHookActions:[]} K3sOptions:{ExtraServerArgs:[[--disable=servic
elb --disable=traefik]] ExtraAgentArgs:[[]]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest:}} Env:[] Registries:{Use:[] Create:false Config:}}
==========================
DEBU[0000] Port Exposure Mapping didn't specify hostPort, choosing one randomly...
DEBU[0000] Got free port for Port Exposure: '62843'
DEBU[0000] ========== Merged Simple Config ==========
{TypeMeta:{Kind: APIVersion:} Name: Servers:1 Agents:3 ExposeAPI:{Host: HostIP:0.0.0.0 HostPort:62843} Image:docker.io/rancher/k3s:v1.20.2-k3s1 Network: ClusterToken: Volumes:[] Ports:[] Labels:[]
Options:{K3dOptions:{Wait:true Timeout:0s DisableLoadbalancer:false DisableImageVolume:false NoRollback:false PrepDisableHostIPInjection:false NodeHookActions:[]} K3sOptions:{ExtraServerArgs:[[--di
sable=servicelb --disable=traefik]] ExtraAgentArgs:[[]]} KubeconfigOptions:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true} Runtime:{GPURequest:}} Env:[] Registries:{Use:[] Create:false Con
fig:}}
==========================
DEBU[0000] ===== Cluster Config =====
&{TypeMeta:{Kind: APIVersion:} Cluster:{Name:k3s-default Network:{Name: External:false} Token: Nodes:[0xc000533e00 0xc0000ca000 0xc0000ca280 0xc0000ca3c0] InitNode:<nil> ExternalDatastore:<nil> Kub
eAPI:0xc00052d3c0 ServerLoadBalancer:0xc000533cc0 ImageVolume:} ClusterCreateOpts:{PrepDisableHostIPInjection:false DisableImageVolume:false WaitForServer:true Timeout:0s DisableLoadBalancer:false
K3sServerArgs:[[--disable=servicelb --disable=traefik]] K3sAgentArgs:[[]] GPURequest: NodeHooks:[] GlobalLabels:map[app:k3d] GlobalEnv:[] Registries:{Create:<nil> Use:[] Config:<nil>}} KubeconfigOp
ts:{UpdateDefaultKubeconfig:true SwitchCurrentContext:true}}
===== ===== =====
DEBU[0000] '--kubeconfig-update-default set: enabling wait-for-server
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-k3s-default'
INFO[0000] Created volume 'k3d-k3s-default-images'
INFO[0001] Creating node 'k3d-k3s-default-server-0'
DEBU[0001] Created container k3d-k3s-default-server-0 (ID: c9c9f769527ae12adafa9357d78bbaadbdd3f942ea8cabb26a9368b7b220cc08)
DEBU[0001] Created node 'k3d-k3s-default-server-0'
INFO[0001] Creating node 'k3d-k3s-default-agent-0'
DEBU[0001] Created container k3d-k3s-default-agent-0 (ID: f208bf1b616964e05373a072046a86fb84b7e37e2e8c5daebf9a7f3bc0b65e73)
DEBU[0001] Created node 'k3d-k3s-default-agent-0'
INFO[0001] Creating node 'k3d-k3s-default-agent-1'
DEBU[0001] Created container k3d-k3s-default-agent-1 (ID: 75ce4577595f38ec2d49e240c7dfffc25fb8eac6ba0edc00bcdf8f04d8bdd056)
DEBU[0001] Created node 'k3d-k3s-default-agent-1'
INFO[0001] Creating node 'k3d-k3s-default-agent-2'
DEBU[0001] Created container k3d-k3s-default-agent-2 (ID: 24f1166d34a00472fc91e7dd07b93ae37e764e90723d657281e4f80fc7c841da)
DEBU[0001] Created node 'k3d-k3s-default-agent-2'
INFO[0001] Creating LoadBalancer 'k3d-k3s-default-serverlb'
DEBU[0001] Created container k3d-k3s-default-serverlb (ID: 6e3abfa12f57264ba494b15a4e062aada5f6d4df779b67b4ad8f84f0a9b56858)
DEBU[0001] Created loadbalancer 'k3d-k3s-default-serverlb'
INFO[0001] Starting cluster 'k3s-default'
INFO[0001] Starting servers...
DEBU[0001] Node k3d-k3s-default-server-0 Start Time: 2021-02-08 23:25:22.217163 +0800 CST m=+1.558799790
INFO[0001] Starting Node 'k3d-k3s-default-server-0'
DEBU[0002] Waiting for node k3d-k3s-default-server-0 to get ready (Log: 'k3s is up and running')
DEBU[0008] Finished waiting for log message 'k3s is up and running' from node 'k3d-k3s-default-server-0'
INFO[0008] Starting agents...
DEBU[0008] Node k3d-k3s-default-agent-0 Start Time: 2021-02-08 23:25:29.163319 +0800 CST m=+8.504782476
INFO[0008] Starting Node 'k3d-k3s-default-agent-0'
DEBU[0009] Waiting for node k3d-k3s-default-agent-0 to get ready (Log: 'Successfully registered node')
DEBU[0021] Finished waiting for log message 'Successfully registered node' from node 'k3d-k3s-default-agent-0'
DEBU[0021] Node k3d-k3s-default-agent-1 Start Time: 2021-02-08 23:25:42.399839 +0800 CST m=+21.740972033
INFO[0021] Starting Node 'k3d-k3s-default-agent-1'
DEBU[0022] Waiting for node k3d-k3s-default-agent-1 to get ready (Log: 'Successfully registered node')
DEBU[0030] Finished waiting for log message 'Successfully registered node' from node 'k3d-k3s-default-agent-1'
DEBU[0030] Node k3d-k3s-default-agent-2 Start Time: 2021-02-08 23:25:50.883821 +0800 CST m=+30.224741504
INFO[0030] Starting Node 'k3d-k3s-default-agent-2'
DEBU[0030] Waiting for node k3d-k3s-default-agent-2 to get ready (Log: 'Successfully registered node')
DEBU[0038] Finished waiting for log message 'Successfully registered node' from node 'k3d-k3s-default-agent-2'
INFO[0038] Starting helpers...
DEBU[0038] Node k3d-k3s-default-serverlb Start Time: 2021-02-08 23:25:58.8086 +0800 CST m=+38.149322857
INFO[0038] Starting Node 'k3d-k3s-default-serverlb'
DEBU[0038] Waiting for node k3d-k3s-default-serverlb to get ready (Log: 'start worker processes')
DEBU[0039] Finished waiting for log message 'start worker processes' from node 'k3d-k3s-default-serverlb'
INFO[0039] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
DEBU[0039] Executing command '[sh -c nslookup host.docker.internal]' in node 'k3d-k3s-default-server-0'
DEBU[0040] Exec process in node 'k3d-k3s-default-server-0' exited with '0'
DEBU[0040] Hostname 'host.docker.internal' -> Address '192.168.65.2'
DEBU[0040] Adding extra host entry '192.168.65.2 host.k3d.internal'...
DEBU[0040] Executing command '[sh -c echo '192.168.65.2 host.k3d.internal' >> /etc/hosts]' in node 'k3d-k3s-default-server-0'
DEBU[0040] Exec process in node 'k3d-k3s-default-server-0' exited with '0'
DEBU[0040] Executing command '[sh -c echo '192.168.65.2 host.k3d.internal' >> /etc/hosts]' in node 'k3d-k3s-default-agent-0'
DEBU[0041] Exec process in node 'k3d-k3s-default-agent-0' exited with '0'
DEBU[0041] Executing command '[sh -c echo '192.168.65.2 host.k3d.internal' >> /etc/hosts]' in node 'k3d-k3s-default-agent-1'
DEBU[0041] Exec process in node 'k3d-k3s-default-agent-1' exited with '0'
DEBU[0041] Executing command '[sh -c echo '192.168.65.2 host.k3d.internal' >> /etc/hosts]' in node 'k3d-k3s-default-agent-2'
DEBU[0042] Exec process in node 'k3d-k3s-default-agent-2' exited with '0'
DEBU[0042] Executing command '[sh -c echo '192.168.65.2 host.k3d.internal' >> /etc/hosts]' in node 'k3d-k3s-default-serverlb'
DEBU[0042] Exec process in node 'k3d-k3s-default-serverlb' exited with '0'
DEBU[0042] Executing command '[sh -c test=$(kubectl get cm coredns -n kube-system --template='{{.data.NodeHosts}}' | sed -n -E -e '/[0-9\.]{4,12}\s+host\.k3d\.internal$/!p' -e '$a192.168.65.2 host.
k3d.internal' | tr '\n' '^' | busybox xargs -0 printf '{"data": {"NodeHosts":"%s"}}'| sed -E 's%\^%\\n%g') && kubectl patch cm coredns -n kube-system -p="$test"]' in node 'k3d-k3s-default-server-0'

DEBU[0044] Exec process in node 'k3d-k3s-default-server-0' exited with '0'
INFO[0044] Successfully added host record to /etc/hosts in 5/5 nodes and to the CoreDNS ConfigMap
INFO[0044] Cluster 'k3s-default' created successfully!
INFO[0044] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
DEBU[0044] Updating default kubeconfig with a new context for cluster k3s-default
DEBU[0045] Setting new current-context 'k3d-k3s-default'
DEBU[0045] Wrote kubeconfig to '/Users/rainflying/.kube/config'
INFO[0045] You can now use it like this:
kubectl config use-context k3d-k3s-default
kubectl cluster-info

% docker inspect k3d-k3s-default-server-0

[
    {
        "Id": "c9c9f769527ae12adafa9357d78bbaadbdd3f942ea8cabb26a9368b7b220cc08",
        "Created": "2021-02-08T15:25:21.833649Z",
        "Path": "/bin/k3s",
        "Args": [
            "server",
            "[--disable=servicelb",
            "--disable=traefik]",
            "--tls-san",
            "0.0.0.0"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 26745,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-02-08T15:25:22.8330743Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:1b02adf07426903bad31c9de2359725a433ae7648b9e33c0fa8778d3dd70b681",
        "ResolvConfPath": "/var/lib/docker/containers/c9c9f769527ae12adafa9357d78bbaadbdd3f942ea8cabb26a9368b7b220cc08/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/c9c9f769527ae12adafa9357d78bbaadbdd3f942ea8cabb26a9368b7b220cc08/hostname",
        "HostsPath": "/var/lib/docker/containers/c9c9f769527ae12adafa9357d78bbaadbdd3f942ea8cabb26a9368b7b220cc08/hosts",
        "LogPath": "/var/lib/docker/containers/c9c9f769527ae12adafa9357d78bbaadbdd3f942ea8cabb26a9368b7b220cc08/c9c9f769527ae12adafa9357d78bbaadbdd3f942ea8cabb26a9368b7b220cc08-json.log",
        "Name": "/k3d-k3s-default-server-0",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "k3d-k3s-default-images:/k3d/images"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=disable"
            ],
            "Tmpfs": {
                "/run": "",
                "/var/run": ""
            },
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": null,
            "ReadonlyPaths": null,
            "Init": true
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/2cc6c1deedc52374011e320c8d5f8160b38558d764e026f3cc6dcce0af26d14b-init/diff:/var/lib/docker/overlay2/2c2eb212c733e319f49117498fcd825b3e0bedcbf269ce251a67aa593de4f07c/diff:/var/lib/docker/overlay2/0e362da0d79e7c1dddbf2a06d0a7fb84ffacddf4deab0d3d62893670ab8f92e4/diff:/var/lib/docker/overlay2/d7f7db0e595ea6b8953548250bdb4b02a335add79779ee401c71760697a9086a/diff",
                "MergedDir": "/var/lib/docker/overlay2/2cc6c1deedc52374011e320c8d5f8160b38558d764e026f3cc6dcce0af26d14b/merged",
                "UpperDir": "/var/lib/docker/overlay2/2cc6c1deedc52374011e320c8d5f8160b38558d764e026f3cc6dcce0af26d14b/diff",
                "WorkDir": "/var/lib/docker/overlay2/2cc6c1deedc52374011e320c8d5f8160b38558d764e026f3cc6dcce0af26d14b/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "k3d-k3s-default-images",
                "Source": "/var/lib/docker/volumes/k3d-k3s-default-images/_data",
                "Destination": "/k3d/images",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "0aae0f32da5412349c2adba72dee9a8044b6ea2849f0a9f9ed9bf9675164334d",
                "Source": "/var/lib/docker/volumes/0aae0f32da5412349c2adba72dee9a8044b6ea2849f0a9f9ed9bf9675164334d/_data",
                "Destination": "/var/lib/cni",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "10dbc12f9015a5ef9999a7aa8b514324f4315e46864aa3739f0eb7da4e991419",
                "Source": "/var/lib/docker/volumes/10dbc12f9015a5ef9999a7aa8b514324f4315e46864aa3739f0eb7da4e991419/_data",
                "Destination": "/var/lib/kubelet",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "dcb6fa227774b50768d19922e0da0c4dfe69e5d18123e7d83a3a5b607da3e52c",
                "Source": "/var/lib/docker/volumes/dcb6fa227774b50768d19922e0da0c4dfe69e5d18123e7d83a3a5b607da3e52c/_data",
                "Destination": "/var/lib/rancher/k3s",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "45add5c7958301ce9e5e2fa34d615be812ab3cce831be40d5448450b129be521",
                "Source": "/var/lib/docker/volumes/45add5c7958301ce9e5e2fa34d615be812ab3cce831be40d5448450b129be521/_data",
                "Destination": "/var/log",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "k3d-k3s-default-server-0",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "K3S_TOKEN=SrKEvnOuywdAMjoJKiqw",
                "K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux",
                "CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml"
            ],
            "Cmd": [
                "server",
                "[--disable=servicelb",
                "--disable=traefik]",
                "--tls-san",
                "0.0.0.0"
            ],
            "Image": "docker.io/rancher/k3s:v1.20.2-k3s1",
            "Volumes": {
                "/var/lib/cni": {},
                "/var/lib/kubelet": {},
                "/var/lib/rancher/k3s": {},
                "/var/log": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "/bin/k3s"
            ],
            "OnBuild": null,
            "Labels": {
                "app": "k3d",
                "k3d.cluster": "k3s-default",
                "k3d.cluster.imageVolume": "k3d-k3s-default-images",
                "k3d.cluster.network": "k3d-k3s-default",
                "k3d.cluster.network.external": "false",
                "k3d.cluster.network.id": "8ffc659d0e8423bbfd1ec051a9bf582f847073b2df24a3d49f012a50248a27b7",
                "k3d.cluster.token": "SrKEvnOuywdAMjoJKiqw",
                "k3d.cluster.url": "https://k3d-k3s-default-server-0:6443",
                "k3d.role": "server",
                "k3d.server.api.host": "0.0.0.0",
                "k3d.server.api.hostIP": "0.0.0.0",
                "k3d.server.api.port": "62843",
                "org.label-schema.build-date": "2021-01-14T23:54:43Z",
                "org.label-schema.schema-version": "1.0",
                "org.label-schema.vcs-ref": "1d4adb0301b9a63ceec8cabb11b309e061f43d5f",
                "org.label-schema.vcs-url": "https://github.com/k3s-io/k3s.git"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "caa18d7ed1c0d0b2405834dce1f517027b6c87d7d10d771341bfe6520208a29d",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/caa18d7ed1c0",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "k3d-k3s-default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "c9c9f769527a",
                        "k3d-k3s-default-server-0"
                    ],
                    "NetworkID": "8ffc659d0e8423bbfd1ec051a9bf582f847073b2df24a3d49f012a50248a27b7",
                    "EndpointID": "3e65e49de66565dcb761baa72e7dbc33b6740ed1daefef09491bcbbe748d33bf",
                    "Gateway": "172.19.0.1",
                    "IPAddress": "172.19.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:13:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

% kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-598b589c46-n4z2v   1/1     Running   0          9m22s

% kubectl expose deployment nginx --type LoadBalancer --port 80
service/nginx exposed

% kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-598b589c46-n4z2v   1/1     Running   0          9m37s
svclb-nginx-dclr7        0/1     Pending   0          3s
svclb-nginx-5jddl        0/1     Pending   0          3s
svclb-nginx-r4zdw        0/1     Pending   0          3s
svclb-nginx-f58m9        0/1     Pending   0          3s

By the way, I've created a tunnel on my mac using tools from https://github.com/AlmirKadric-Published/docker-tuntap-osx.
Is it related to this issue ?

% ip r
default via 192.168.0.1 dev en0
10.0.75.0/30 dev tap1  scope link
127.0.0.0/8 via 127.0.0.1 dev lo0
127.0.0.1/32 via 127.0.0.1 dev lo0
169.254.0.0/16 dev en0  scope link
172.18.0.0/16 via 10.0.75.2 dev tap1
192.168.0.0/24 dev en0  scope link
192.168.0.1/32 dev en0  scope link
192.168.0.100/32 dev en0  scope link
224.0.0.0/4 dev en0  scope link
255.255.255.255/32 dev en0  scope link

% ip addr show dev tap1
tap1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        ether 4e:9c:fa:e6:2e:72
        inet 10.0.75.1/30 brd 10.0.75.3 tap1

[iwilltry42]

@RainFlying facepalming myself right now 🤦‍♂️
Actually this was hit fixed in k3d v4.1.1 and I didn't realize that you're on v4.1.0..

[RainFlying]

@RainFlying facepalming myself right now 🤦‍♂️
Actually this was hit fixed in k3d v4.1.1 and I didn't realize that you're on v4.1.0..

My fault. I installed 4.1.1 yesterday but forgot to enable it. Thought the latest release was 4.1.0 today.
Sorry for wasting your time.
Retried with k3d v4.1.1. Yes, the issue has been fixed.

[iwilltry42]

No worries, I also didn't get it 😁