Feature request: Better policy construct to match on apiserver connectivity

[joestringer]

The Kubernetes apiserver typically lives on the kubernetes master nodes, and as such is addressed by the IPs of those nodes. When traffic to or from those nodes is handled at a peer Cilium-managed pod, the identity of those IPs for policy purposes is typically established as remote-node (or host if the pod is scheduled on the same node).

To allow such communication with the apiserver via network policy, typically today the way is to use Entities-based policies to match on the traffic, allowing the pod to communicate with all nodes in the cluster (just the nodes, not the pods on those nodes). This can be limited to specific ports and managed via L7 policies to restrict the pod's ability to communicate, but that traffic may reach any node in the cluster, not just to the apiserver.

If the user attempts to only use Kubernetes Network Policies to allow the traffic, there is no good mechanism as the K8s NP does not provide an abstract representation of the apiserver or nodes in the cluster, it only has IP-based structures for allowing such traffic. This forces the user to encode addressing schemes into their network policies, which is not ideal as it requires manual intervention if the nodes are changed or a similar policy is applied to another cluster (staging vs. prod, primary vs. secondary). Furthermore, as the identity for the traffic is established as remote-node, #12277 prevents matching on the traffic using IP-based policies.

In some environments, typically where the Kubernetes control plane is managed by a provider, the apiserver may alternatively live outside the kubernetes cluster entirely, in which case the above approach does not work, but instead users may use toServices, toCIDR or fromCIDR matches to match on the traffic. This works, but requires the user to consider where their control plane is deployed and adjust the policy based on this. An ideal solution would allow the policy to understand how to allow traffic to/from the apiserver regardless of its location.

If there were unique identities per node in the cluster, this would make it easier to match specifically on apiserver traffic from network policy; we could likely come up with either new abstract constructs (eg an entity) to represent the apiserver in policy, or handle the traffic through toServices constructs to allow pods to contact the apiserver via the kubernetes service, or even potentially match on IP.

If you are interested in this feature, please react with 👍 to this issue. This helps the core Cilium team to understand demand and prioritize work.

[joestringer]

Another option is to disable the remote-node identity by setting enable-remote-node-identity to false and transmit-host-src to true, then match on the traffic by IP.

If solving this issue requires modifying the identity of live nodes in the cluster, then close attention must be paid to upgrade impact in a similar way to the instructions laid out in the v1.7 upgrade notes for enable-remote-node-identity option.

[pchaigno]

To some extent it is also possible to restrict access to the kube-api server using the host firewall (beta feature). A host policy can be loaded on all nodes to deny ingress access from cluster pods and allow other accesses (world, remote nodes, etc.). Then, a second policy can be loaded on the master node only, to allow access to the kube-api server.

If using the host firewall, care must be taken to ensure required ports are open. It is also strongly recommended to validate the policy in audit mode because an incorrect policy can brick the cluster.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.