CNI: Proxy skip ports configuration are ignored

[ihcsim]

Bug Report

What is the issue?

In a CNI setup, init containers that try to make outbound connection will fail due to a timing issue between the CNI plugin modifying the pod iptables, and the proxy container coming up. The current workaround is to skip the outbound ports of the init containers. However, this only works if the skip port configuration is provided during the CNI installation, implying that all workloads will skip these ports.

The linkerd inject --skip-[in|out]bound-ports and its annotation equivalence, are currently ignored by the CNI plugin.

How can it be reproduced?

1. Install the Linkerd CNI plugin and control plane
2. Deploy a workload with an init container that performs outbound connection:

$ cat <<EOF | k apply -f -
kind: Deployment
apiVersion: apps/v1
metadata:
  name: curl
spec:
  selector:
    matchLabels:
      app: curl
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: curl
    spec:
      initContainers:
      - name: curl-init
        image: appropriate/curl
        command: ["/bin/sh", "-c"]
        args: ["curl", "google.com"]
      containers:
      - name: curl
        image: appropriate/curl
        command: ["sleep", "3600"]

EOF

The init container should fail to come up due to connection refused errors.

3. Re-deploy the Deployment with the config.linkerd.io/skip-outbound-ports: 80,443 annotation. The init container will still fail to come up.

Possible solution

Update the CNI plugin to check for skip ports annotations.

[chris-vest]

I am also finding this issue, version 2.8.1 and latest CNI.

linkerd check ==> Status check results are √

Setting the config.linkerd.io/skip-outbound-ports: 443 annotation as suggested in a number of issues (e.g. #4788) does not solve the initContainer connectivity to, for example, the Kubernetes API (Grafana init container) or Vault (vault-agent init container).

[Pothulapati]

@ihcsim I was trying to take this up! But wonder how we can make CNI plugin know about the skip port annotations, as its installed before injection and I dont see any touchpoints during inject and CNI to make this possible. Do you have any suggestions/solutions here?

[alpeb]

@Pothulapati that touchpoint could be cmdAdd() that gets called whenever a container is created. We have the pod config in there so we can easily extract the annotations and override the iptables rules. But the logic there is such that the rules are set only after the proxy has been injected, so I don't understand why the init containers' outbound connections are being blocked 🤷‍♂️

[ihcsim]

the logic there is such that the rules are set only after the proxy has been injected,

@alpeb I think the rules are added after the pod is mutated, but before it's run. Then when all the containers start up, the rules would have been added.

[alpeb]

Oh that'd make sense. So pulling the annotations in cmdAdd() and overriding them should do the trick, even for the init containers 👍