Support portforward for kata and gVisor

[Random-Liu]

We recently added portforward support for windows https://github.com/containerd/cri/blob/master/pkg/server/sandbox_portforward_windows.go. It exec into the pause container, and runs wincat.exe to do the bidirectional data redirection.

We can generalize the solution, and use it to support kata and gVisor, and even replace the current runc support to get rid of the dependency of socat on the host.

Option 1: Keep the socat implementation for runc, and support this for kata or gVisor.

We need to introduce 2 options PortforwardExecCmd and SandboxImage into runtime config https://github.com/containerd/cri/blob/master/pkg/config/config.go#L31.

And the CRI plugin will exec to run the PortforwardExecCmd instead of running socat if the 2 options above are configured for a runtime.

Option 2: Use this approach for everything.

This one requires changing the default sandbox image.

We are using k8s.gcr.io/pause:3.1 right now.

We need to:

1. Either maintain our own pause image;
2. Or work with Kubernetes upstream to add a socat implementation into the pause image.

Moving socat implementation into the pause image will get rid of the socat dependency for all Kubernetes users, and make it possible to support kata and gVisor, and windows is already doing that. I think it is worth doing.

NOTE that was previously discussed in https://docs.google.com/presentation/d/1x1B2_DFZ9VI2E_-pB2pzeYKtUxrK4XLXd7I0JWD1Z40/edit#slide=id.p.

One useful discussion there about this approach:

This makes the pause image larger, and increase the memory footprint when portforward runs.

However, pause image is only pulled once, and all pause containers share the same image, I don't see it introduce too much overhead.

As for memory footprint, if user doesn't use portforward, they won't be affected. If they are, the overall memory foot print is no difference with today, the extra memory usage is just moved into the pause container. The good thing is that the extra memory usage will be charged now; the bad thing is that this destabilize the pause container.

One option is that we keep pause process with original oom score, and set higher oom score adj for other utility exec processes.

@fvoznika @egernst @gnawux

[Random-Liu]

@ianlewis pointed out that the option 3 is also worth evaluating.

Although it does look more complex than option 2, especially the container lifecycle part, we need to hide it from kubelet, and manage its lifecycle ourselves. However, it is doable, and most runtime agnostic, because no sandbox container is required to exist.

I think the original concern about option 3 is about the resource management, but given that Kubernetes already supports Ephemeral Containers, we just need to follow the same principle - lowest priority, no resource guarantee at all.

[ianlewis]

I think my observation at the time was that port-forward seems like just another use case of running debug containers and that the EphemeralContainers API was created for that purpose. I'm not sure that the current API allows for something like port forwarding but if users could run something like kubectl [alpha] debug .... with some options to achieve the same thing as kubectl port-forward then that feels like a better design to me than adding socat to the pause container or otherwise.

[ianlewis]

I expanded on some options in this issue:
kubernetes/enhancements#1846

Would appreciate some feedback on the different options.

[egernst]

draft implementation on kata side: kata-containers/kata-containers#5979

[antonr-p2p]

Also some work was done here, but it's closed: #4814

[milantracy]

is the issue actively being worked on?

[ianlewis]

is the issue actively being worked on?

Not to my knowledge on the containerd side. The last patch or POC was my PR #4814. This was rejected initially by the containerd team because they wanted it to be based on a better internal API or set of interfaces.

I think someone could work on it now that the Sandbox API is now done and this could be modeled after that as an extension to the shim API. Though, I'm not exactly sure how that landed.

[nia-potato]

Hi containerd team, since this issue was closed because the team wanted a better internal API to do this.

is there a new issue number we are tracking the progress of this for?

we use gvisor extensively, but the only issue that is blocking us to do a full migration is due to this issue of port-forwarding not supported after gvisor is enabled.