port forward not working properly on mac

[prabalsharma]

Actual Behavior

Sometimes wordpress url cannot be reached on Mac and Linux.

Steps to Reproduce

helm repo add bitnami https://charts.bitnami.com/bitnami
 
helm install --wait --timeout=20m wordpress bitnami/wordpress --set=service.type=NodePort --set=volumePermissions.enabled=true --set=mariadb.volumePermissions.enabled=true
 
kubectl get service/wordpress

export NODE_PORT=$(kubectl get --namespace default -o jsonpath="{.spec.ports[0].nodePort}" services wordpress)

Try to access UI "http://localhost:$NODE_PORT"

Result

UI will not be reachable

Expected Behavior

Should be able to reach UI on NodePort

Additional Information

Can reach UI using node IP http://$NODE_IP:$NODE_PORT/
export NODE_IP=$(kubectl get nodes --namespace default -o jsonpath="{.items[0].status.addresses[0].address}")

Rancher Desktop Version

1.4.1-485-g29daa817

Rancher Desktop K8s Version

1.23.8

Which container runtime are you using?

containerd (nerdctl)

What operating system are you using?

macOS

Operating System / Build Version

macOS Catalina

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

No response

Windows User Only

No response

[gunamata]

It looks like it has got something to do with specific Kubernetes versions. On my Win 10 Enterprise, I could repro the issue with the kubernetes version v1.23.8, whereas I could access the application with the version v1.21.0.

[ericpromislow]

So I'm still not an expert in this area, but there's no ingress or LoadBalancer running -- aren't those required to be able to reach an app via localhost ?

[jandubois]

@ericpromislow No, all NodePort services are supposed to be forwarded as well. That's kind of the point of them: they allocate a (random) port on the local "node", which in this case is the VM.

The issue is that the CNI may only create an iptables rule for the port, and we need to match that rule to realize that this port should be forwarded.

[jandubois]

I get the same behaviour on 1.0.1, 1.3.0, 1.4.1, and the latest CI builds.

The difference seems to be in the k3s version: 1.23.6 seems to work correctly, but 1.23.8 doesn't seem to forward the nodeport.

[jandubois]

Automatic NodePort forwarding

• works for 1.21.x up to 1.21.11, fails for 1.21.12
• works for 1.22.x up to 1.22.9, fails for 1.22.10, 1.22.11
• works for 1.23.x up to 1.23.6, fails for 1.23.7, 1.23.8, 1.23.9
• fails for 1.24.1, 1.24.2, 1.24.3

So this problem is due to a change in k3s/k8s and not a regression in Rancher Desktop.

[jandubois]

@brandond pointed to kubernetes/kubernetes#108496 as the reason why newer releases of Kubernetes no longer keep the port open.

Our plan is to create a watcher process inside the VM that will connect to the apiserver and listen to service changes. It will start listening on the TCP4 port when a new service is created, and close the port when the service is gone.

This is very similar to what we already do for WSL2 in https://github.com/rancher-sandbox/rancher-desktop-agent

Ideally the new watcher can be included in this agent, and when the agent runs on macOS or Linux, it will not do the iptables parsing part because that is already being done by Lima.

The code should be written in a way that it can be migrated over to Lima in the future, to become part of the Lima guest-agent.

[mook-as]

Overview of current port forwarding status (Ignoring Kubernetes, as that will need to be changed):

OS	dockerd	containerd
Windows	docker-proxy	CNI / rancher-desktop-guestagent
macOS	docker-proxy	CNI / lima-hostagent

So we'll need to build something, but also keep the existing iptables scanning code (for containerd).

[jandubois]

This issue is a duplicate of #2489; please close both once a fix has been made!

[jandubois]

We will not be able to address this issue in the upcoming 1.5.0 release, but will have a patch release later in August that addresses it.

If possible use an older Kubernetes release that doesn't include the change responsible; see #2593 (comment) for a list.

[mook-as]

rancher-sandbox/rancher-desktop-agent#4 create for rancher-desktop-agent; once that merges, we still changes in RD to

• use the updated agent
• enable kubernetes monitoring as required
• use the agent on lima (we currently don't)