Allow for operation on clusters that restrict NET_ADMIN

[grampelberg]

Feature Request

What problem are you trying to solve?

Many production clusters are locked down and do not allow arbitrary pods to have cap_net_admin. This makes it, unfortunately, impossible to use linkerd in these environments.

How should the problem be solved?

It would be nice to have an option (addon?) that does the iptables configuration outside of a pod's initContainer. This would allow an operator with elevated privileges to add the solution to the cluster and have arbitrary pods have minimal privileges.

A good solution would be creating a CNI plugin that does the iptables setup. A DaemonSet would be added to the cluster which adds the CNI plugin to the set. Each time a pod is scheduled, the iptables rules would be added as part of network setup. This allows for them to be in place when the pod starts up (there might be some oddities with initContainers?).

Take a look at istio/cni for some good previous work.

Any alternatives you've considered?

• DaemonSet controller that updates iptables - Obviously the easiest solution, there are real sync problems and issues with pods starting that don't have rules in place (making policy a little difficult).
• HTTP_PROXY options - Would only work for outgoing traffic, which isn't great.

How would users interact with this feature?

This should be optional and not default. It would be an awesome use of addons:

linkerd add secure-cluster-helper

There are definitely some big open questions about the interactions between this and things like the auto-inject feature.

[mmack]

Great to see this picked up! If you need testers let me know...

In my opinion, even if it is not required, skipping root images and avoiding cap_net_admin priviledges is always a good thing and should be the default. Or are there any downsides other than deploying a deamonset for management?

[grampelberg]

@mmack there are definitely tradeoffs. After digging into this some more, it looks like going the CNI route is more sustainable than a daemonset controller (for obvious sync reasons). Unfortunately, not every environment lets you add a new CNI plugin.

Ideally, this would be the default though, just need to make sure it works in most cases and has clear guard rails. Some check integration would help quite a bit.

[mmack]

@grampelberg so the plan is to add a second CNI? My primary is calico atm...

[grampelberg]

@mmack that looks like the best solution I can find so far. You add a new CNI plugin to the chain that simply sets the iptables rules up (and cleans them up on teardown).

[grampelberg]

CNI plugin

• Base on containernetworking/cni.
• Hook ADD requests and use (or update) the existing iptables code.
• Hook DELETE requests and remove the iptables rules.

Some things to pay attention to:

• kube-system does reconciliation for most components. They won't be able to have a sidecar proxy and therefore should not have iptables setup.

Potential solution:

• Key the addition of iptables rules off metadata (a linkerd.io/setup-iptables: true label?).

DaemonSet for installation

Before including this as part of the install manifest, keep a separate YAML to do install. An example of how to do this can be found by looking at calico.

linkerd install modifications

linkerd install --no-init-container

linkerd inject modifications

linkerd inject --no-init-container

Auto-inject modifications

Just make --no-init-container work with --proxy-auto-inject. At the moment, this is a separate code path from inject. It would be amazing to tackle this at the same time as #1748 .

Control plane integration

Integrate into the install output as an option. While it is sufficient to simply add a flag to install, it would be great to eventually:

• Tackle Introduce the concept of addons #1891 and bundle the DaemonSet as an addon.
• Tackle Fetch inject configuration from control plane #1999 and have inject default to --no-init-container.