Feature request: CNI initcontainer full proxy

[tspearconquest]

What problem are you trying to solve?

In #8100 I discussed how to work around an issue with initContainers that need functioning network connectivity during the init phase of the pod's lifecycle in Prometheus, and the workaround currently is to skip the outbound ports that need to communicate during this phase so that the proxy is bypassed since it isn't up until later in the lifecycle. Since the CNI forwards ports to the proxy outside of the lifecycle of any individual kubernetes resource, containers are unable to connect until the proxy is running (unless you skip outbound ports).

Unfortunately this means that for all of the containers in the pod, we lose that visibility, and for port 443 outbound, that is suboptimal.

This is additionally evidenced by the need for the proxy await flag for various apps, however I think this feature won't impact that issue in any way.

How should the problem be solved?

What I was thinking is that it would be great if we can have a running proxy in the init phase which can be told to shutdown once the work that requires it is completed.

This way, we have a proxy for the init containers and it shuts down, and then the regular containers starts up with their own proxy that stays up for the life of the pod.

Any alternatives you've considered?

We're currently skipping the outbound port 443, which defeats the purpose of using linkerd.

How would users interact with this feature?

Potentially a new annotation which tells the proxy injector to inject a proxy in the initContainers of the resource.

Would you like to work on this feature?

No response

[tamilhce]

The CNI plugin creates iptables rules to route all the egress traffic through linkerd-proxy sidecar.
The linkerd-porxy sidecar container will be running only after the init containers are completed.
This blocks the init container outbound traffic completely. we can add a check to the Linkerd cni plugin : https://github.com/linkerd/linkerd2/blob/main/cni-plugin/main.go#L191
to add the IPtables rules only after the init containers are in completed/not running state. I will go ahead make those changes. if it make sense. Let me know your comments

[tspearconquest]

Hello,

I believe with the CNI plugin running as a daemonset with hostNetwork: true that it is setting the rules on the host
network namespace. Will it work for this use case?

[tamilhce]

I think yes, it works for the hostNetwork as well. The only change with the proposed approach is, in case of Pod with init containers, we are applying the Linkerd specific Iptables rules only after the init container is completed.

[olix0r]

@tamilhce you're welcome to try this approach, but I'm skeptical that it will work as you hope. My expectation is that the pod won't actually start until the CNI is initialized. If you do try to defer this configuration until after init containers run, I'm not sure how you can guarantee that this initialization happens before the application pod starts. I guess you'd need to add some init container that blocks until the iptables are configured? There's a lot of moving pieces here.

I think this warrants some experimentation, but we'd need a more detailed understanding of how this would work before we can say what we'll actually merge.

[mamjong]

Any update on this by any chance?
We are also running into this issue, we'd like to use the CNI-plugin, but we also have init containers that need a network connection.

[mateiidavid]

@mamjong I'd be curious if @tamilhce has managed to give their poc a try. I'm also skeptical that it would work, but if it does, I think we're all for having it in. My skepticism comes from the fact that CNI plugins, in general, are used to create the sandbox and network namespace for a pod. Like mentioned above, I'm not sure there's a way to halt the CNI plugin execution while containers run. Containers need to have their network initialized before being started. Plugins are also generally chained and executed in-order, I'm not sure we have any guarantees that this behaviour wouldn't affect any plugins that run after our own CNI plugin.

There's a lot to unpack here.

[tamilhce]

I apologize for the delay in responding. I was wrong in my assumption. I did not find a way to halt the CNI Plugin, as @mamjong mentioned.
The idea of enabling sidecar containers before normal containers and shutting them down after other containers are done is being discussed here kubernetes/enhancements#753. If you have any other suggestions, I will give it a try

[tspearconquest]

Hello,

In my original configuration before we enabled CNI, the grafana initContainer mentioned in #8100 would start up properly without having to have the annotation config.linkerd.io/skip-outbound-ports: 443. In that configuration, there was a linkerd-proxy-init container which was injected and that ran before the other initContainers.

Since linkerd-proxy-init is both a proxy itself and sets up the CNI, and is available in the pod's context during init, all of the the grafana initContainers in that grafana deployment would start up without requiring the annotation above.

I'm not clear from the responses so far on why the linkerd-proxy-init container can't be injected into the deployment to act as a proxy during init even with CNI enabled, aside from the fact that enabling the CNI in the config simply disables the proxy init.

So in other words, I don't understand why a new boolean pod annotation couldn't be created to inject the same linkerd-proxy-init container but have that simply act as a proxy during the pod's init phase without doing the firewall config in the pod context? The CNI handled the firewall config in the node, so the the traffic is already being routed to the right place by the kernel, but there is nothing listening to proxy it during pod init, so if we can simply have an initContainer that runs the proxy, I think it should fulfill the requirement.

[olix0r]

Since linkerd-proxy-init is both a proxy itself and sets up the CNI

linkerd-proxy-init is not a proxy. The only thing it does is run iptables commands. See https://github.com/linkerd/linkerd2-proxy-init for sources.

[tspearconquest]

Ah, ok. I realize I misinterpreted the fact that the init containers were working before as being due to the proxy init container, but it seems maybe that the init containers simply were running before the proxy init applied the iptables rules, which allowed it to work but the traffic wasn't visible in linkerd. Thanks for clarifying that :)

[olix0r]

From https://kubernetes.io/docs/concepts/workloads/pods/init-containers/:

Init containers are exactly like regular containers, except:

• Init containers always run to completion.
• Each init container must complete successfully before the next one starts.

So, it's unfortunately not possible for an init container to start a proxy that is usable by other init containers.

[wc-s]

As a workaround, instead of skip-outbound-ports, is it possible to "skip outbound hostnames" or something like that? This way we can allow the init containers to do their thing, while still being able to proxy normal containers' real port 443 traffic. Or maybe the skip-outbound-ports can target just the specific init containers? I admit to not being very familiar with iptables so I don't know if these options are feasible.

[adleong]

Hi @wc-s. Unfortunately, this is not possible. iptables works at the IP layer and can't see things like hostnames.

[wc-s]

@adleong got it thanks. Yea kinda suspected that'd be the case.

Then it does seem like we must either skip-outbound-ports, or use the linkerd init container (and somehow make sure that the it is the last init container to be run).

[tspearconquest]

Kubernetes doesn't guarantee startup order, so skip-outbound-ports is the only option.

[wc-s]

k8s runs the init containers sequentially in the order in which they are defined in the manifest. So I think the linkerd proxy injector can try to insert the linkerd init container as the last init container?

For comparison, hashicorp's vault agent injector has an annotation "vault.hashicorp.com/agent-init-first", that makes it try to inject the vault init container as the first init container. (https://www.vaultproject.io/docs/platform/k8s/injector/annotations)

[kleimkuhler]

For what it's worth Linkerd already inserts proxy-init as the last init container. Notice the final - in path.

Keep in mind this doesn't guarantee proxy-init is the final container, as the Pod may go through additional webhooks that add other init containers.

[bartwitkowski]

Istio has the same problem with CNI and initContainer and besides the skip-outbound-port or skip-subnets they have third workaround:

"Set the uid of the init container to 1337 using runAsUser. 1337 is the uid used by the sidecar proxy. Traffic sent by this uid is not captured by the Istio’s iptables rule. Application container traffic will still be captured as usual."

Is it possible to do sth similar in Linkerd?

[olix0r]

@bartwitkowski Yes, it should be possible to use the proxy's UID (2102, i believe?) to accomplish the same outcome.

[bartwitkowski]

@olix0r It works like a charm! Just tested it. The Linkerd documentation should be updated because that is the best workaround for this problem.

[wmorgan]

Great to hear @bartwitkowski . Where do you think is the best place to document this? Can you file a linkerd/website issue to help us track it please?

[tspearconquest]

Wow, thanks I wasn't aware of this before!

Just to confirm I understand correctly, I need to add/change runAsUser to match the proxy's UID on my initContainer which isn't related to linkerd in any way? So for example a pod that runs prometheus and has an initContainer, the initContainer in that prometheus pod?

If the initContainer requires to be run as root or some other specific UID for some reason, then this wouldn't work for that case, is that correct?

[olix0r]

If the initContainer requires to be run as root or some other specific UID for some reason, then this wouldn't work for that case, is that correct?

Correct. Also, if you run other non-init containers with this UID, they will not be a part of the mesh.

[bartwitkowski]

Great to hear @bartwitkowski . Where do you think is the best place to document this? Can you file a linkerd/website issue to help us track it please?

Better late then never linkerd/website#1512 ;-).

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.