Unable to connect to host VPN through rancher desktop on Macos

[yevon]

Actual Behavior

When you have a VPN connected in the host, kubernetes nodes are unable to communicate with servers in the VPN. This doesn't happen in windows, this works nicely. Related issue in Lima VM repository:

lima-vm/lima#587

Steps to Reproduce

Connect a VPN on macOS in the host, and try to ping from one of the nodes of the cluster to a computer in the host VPN, it will say "unreachable".

Result

Unreachable hosts within kubernetes hoster via VPN in the host.

Expected Behavior

It should be able to communicate with any computer within the cluster that the host has access to. If I make changes to underlying Lime VM routing tables, those changes could be lost if I update rancher os.

Additional Information

No response

Rancher Desktop Version

1.5.1

Rancher Desktop K8s Version

1.21

Which container engine are you using?

containerd (nerdctl)

What operating system are you using?

macOS

Operating System / Build Version

macOs Monterey 12.0.1

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

No response

Windows User Only

No response

[adamkpickering]

This issue started in a discussion: #2740. I should note that @yevon was using wireguard as their VPN.

[chriscasola]

I am having the same issue, using GlobalProtect VPN.

Edit: adding some more detail

This seems to be because certain hosts are being routed to the network interfaces created by docker for local docker networks created using docker network create.

If I delete my docker networks this issue is resolved.

[jandubois]

Thanks @chriscasola !

@yevon I've just tested the setup you described: I've connected to a remote VPN using Viscosity. Then I've deployed a container on the Rancher Desktop kube cluster.

I started an interactive session inside that container and verified that the name of a remote host resolves. That proves that DNS lookup follows the split-DNS configuration provided by Viscosity.

Then I installed openssh into the container and started an ssh session to the remote machine on the other side of the VPN, and that worked too, showing that packets where routed correctly. It felt a bit slow, but was otherwise working fine.

So I cannot reproduce the problem you are having. Can you provide additional details? Otherwise I don't know what else we can do.

[yevon]

Thanks @chriscasola !

@yevon I've just tested the setup you described: I've connected to a remote VPN using Viscosity. Then I've deployed a container on the Rancher Desktop kube cluster.

I started an interactive session inside that container and verified that the name of a remote host resolves. That proves that DNS lookup follows the split-DNS configuration provided by Viscosity.

Then I installed openssh into the container and started an ssh session to the remote machine on the other side of the VPN, and that worked too, showing that packets where routed correctly. It felt a bit slow, but was otherwise working fine.

So I cannot reproduce the problem you are having. Can you provide additional details? Otherwise I don't know what else we can do.

Hi thanks for testing this! Might be vpn related then, Any special config? Might be due to allowed subnetworks ip mask in the vpn?. I will try to reach the user with the mac for further testing. I will try what @chriscasola suggests also.

[yevon]

Did you activate IP forwarding or set up some nat routes?

[jandubois]

Did you activate IP forwarding or set up some nat routes?

No, I just connected via Viscosity with my OpenVPN profile, and that was it.

[chriscasola]

@jandubois should I file a separate issue for the docker network problem we're having at my company? Docker networks seem to work fine for days/weeks but then all of a sudden none of the containers can reach hosts on the VPN. The only solution is to delete all docker networks and containers and recreate them.

It seems like a routing issue, where connections from within the containers start routing to the local network instead of the VPN network, but I haven't been able to confirm that. Any tips on how to debug would be appreciated.

[chriscasola]

Bumping this again because it's becoming really frustrating to have to delete all my docker networks and containers and recreate them to resolve this issue.

Is there anything I can do to help move this along?

[jandubois]

should I file a separate issue for the docker network problem we're having at my company? Docker networks seem to work fine for days/weeks but then all of a sudden none of the containers can reach hosts on the VPN. The only solution is to delete all docker networks and containers and recreate them.

Yes, please file a separate issue, as that sounds like a different problem.

However, I'm not sure what we can do about it unless we can reproduce the problem.

So restarting Rancher Desktop or even rebooting the host machine does not resolve the problem? You have to delete the networks and containers?

@Nino-K Do you have any ideas?

[Nino-K]

Docker networks seem to work fine for days/weeks but then all of a sudden none of the containers can reach hosts on the VPN. The only solution is to delete all docker networks and containers and recreate them.

@chriscasola when the issue occurs, have you tried inspecting the subnet IP address range that is used by the docker network? to make sure it is not conflicting with the VPN network?

[yevon]

Seems that docker desktop faced sames issues with mac m1 and big sur, docker/for-mac#5322
@jandubois , is your mac an M1 with big sur? I will try some of the workarround they mention on this issue.

[jandubois]

is your mac an M1 with big sur?

No, it is an Intel machine with Catalina. My M1 machine with Big Sur is on the other side of the VPN...

[chriscasola]

Spun off my issue to #3161 although I'm not convinced these are actually different issues.

[chriscasola]

@Nino-K I think you were right about the docker network subnets conflicting with the VPN network. I found this issue in moby while digging around and it seems like I can change the default subnets for docker network create which should solve my issue, will report back.