[ENHANCEMENT] Creative solution for port binding issues?

[ibuildthecloud]

Right now you have to do -p to bind any port you'd wish to access k8s. On linux this isn't much of an issue because you can just access the k3d container IP directly but on docker-for-mac and windows this isn't possible. I'm wondering if we could get creative here. What we could do is deploy a controller that watches hosts ports of pods and then dynamically create docker containers that just forward that one port. For example....

If I can create a service LB in k3s that exposes port 80, what we do is actually create a pod that uses host port 80 and forwards all request to the service LB's cluster IP. We would see that a pod is using host port 80 and deploy a docker container with -p 80:80. In that docker container we would then create an iptables rule that forwards all traffic coming in on 80 and forwards to the k3s_default container. Then VPNKit on docker-for-mac, docker-for-windows will create a forwarding tunnel from the desktop to the new docker container we just created.

The only issue I see with this approach is if you have multiple k3s clusters running. We could do something then like "port offset". So one cluster gets an offset of 0 so port 80 and 443 will be bound to 80 and 443. Then a second cluster would get an offset like 10000 so ports get bound to 10080 and 10443.

WDTY?

[zeerorg]

Instead of dynamically creating docker container, another thing we can do is run the port-watcher in host network, it will watch ports and forward them as required using a proxy server.

About multiple k3s clusters the port offset option seems good.

[ibuildthecloud]

@zeerorg How would the port-watcher work? The basic issue is that on docker-for-mac the VM IP is not routable to the desktop (outside the VM). So creating docker containers is a hack to get VPNKit to create tunnels to the host.

[zeerorg]

One idea is to query the kubernetes api for all "HostPort" services every fixed time interval. and then expose those.

Another idea is to smartly use kubectl port-forward to just skip creating docker containers and instead run a k3d service in the background that handles redirects.

[ibuildthecloud]

We could probably do something really intelligent and do k3d port-forward that dynamically binds and forwards ports. It wouldn't be too hard to write a controller to do that.

[zeerorg]

I'll try a sample implementation of this, just do k3d port-forward and for all HostPort and NodePort binds run kubectl port-forward.

Running a different docker container for every port seems like it could go out of hand if we have lots of ports.
Do you have any adv/disadv for both of these?

Immediate adv that second implementation has is that it could do udp port-forward but it will be harder to implement.

[vincent99]

There's also this directory in the moby VM which directly manipulates the VPNKit port mapping, but it's not clear if there's any way to get to it from outside (-v /var/vpnkit/port:/port tries to find that on the exported list of the Mac):

# cat /var/vpnkit/port/README

Directory of active Instances
-----------------------------

Every active Instance is represented by a file. To shut down an Instance,
remove the file.

To request an additional Instance, make a directory with a unique name,
then open the special file `ctl` inside, and `write` a single string of the
following form:

tcp:<local IP>:<local port>:tcp:<remote IP>:<remote port>
udp:<local IP>:<local port>:udp:<remote IP>:<remote port>
unix:<base64-encoded local path>:unix:<base64-encoded remote path>

Immediately read the file contents and check whether it says:

- `OK <instance details>`: this means the Instance has been configured and
  the details returned to you. For some instance types the server might modify
  the request slightly, for example by choosing a local port number or
  temporary path.
- `ERROR some error message`: this means the Instance creation has failed,
  perhaps some needed resource is still in use.

The directory will be deleted and replaced with a file of the same name.

[zeerorg]

vpnkit port mapping certainly seems the right thing to do, I'll try to scrounge docker windows and mac source code to see how vpnkit is being used.

[ibuildthecloud]

Awesome, automatic port binding would greatly increase the usability.

[ashb]

I don't know where the /var/vpnkit/port mount comes from, but on the mac host there is few unix sockets in ~/Library/Containers/com.docker.docker/Data/

srwxr-xr-x  1 ash  staff    0  3 Apr 16:19 vpnkit.diag.sock
srwxr-xr-x  1 ash  staff    0  3 Apr 16:19 vpnkit.eth.sock
srwxr-xr-x  1 ash  staff    0  3 Apr 16:19 vpnkit.pcap.sock
srwxr-xr-x  1 ash  staff    0  3 Apr 16:19 vpnkit.port.sock

And the code in VPNKit that speakes to it: https://github.com/moby/vpnkit/blob/2b3920dab1da4c5cddeb94f4feb31a18700fefa9/go/pkg/vpnkit/port.go (and other files in that folder)

[iwilltry42]

Any thoughts on this #89 ?
@zeerorg could you find anything about using vpnkit?

[iwilltry42]

Any update on this?

[kslacroix]

@ibuildthecloud Why isn't it possible to access the docker container IP on macos?

[iwilltry42]

@ibuildthecloud Why isn't it possible to access the docker container IP on macos?

Because on MacOS, docker is running inside a VM, so there would have to be a route host -> Docker VM -> container

[kslacroix]

Because on MacOS, docker is running inside a VM, so there would have to be a route host -> Docker VM -> container

Ahh and does it mean that if I run docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' CONTAINER_ID, the resulting IP address only makes sense inside the VM?

[iwilltry42]

Because on MacOS, docker is running inside a VM, so there would have to be a route host -> Docker VM -> container

Ahh and does it mean that if I run docker container inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' CONTAINER_ID, the resulting IP address only makes sense inside the VM?

Yep, to my knowledge that's how it is.