Add support for port ranges in NetworkPolicy with "endPort" field

[mitre-dleung]

Component (Azure NPM or Azure CNI):
Azure CNI

Describe in detail the feature/behavior/change you'd like to see:
Add support for specifying the endPort field in a NetworkPolicy to open a range of ports https://kubernetes.io/docs/concepts/services-networking/network-policies/#targeting-a-range-of-ports

Context:
Numerous Azure services require a large range of ports for communication (on the order of thousands). To isolate pods with NetworkPolicy, a default-deny NetworkPolicy is enforced with a k8s policy management solution, then only the required connections are whitelisted with principal of least privilege.

Problem:
To allow a connection to an azure service outside the k8s cluster that requires a large range of ports to be open, a port entry needs to be defined for every port in the range, resulting in extremely long resource definitions. Especially when these NetworkPolicies are enforced with k8s policy management, there is an insane amount of load put on the kube-apiserver possibly resulting in an unintended denial-of-service on the entire cluster.

Current workarounds:

1. Currently, all ports need to be opened for the destination CIDRs of the Azure service(s) via not specifying ports in the NetworkPolicy, which does not follow principal of least privilege and is not as secure.

2. Create the long NetworkPolicies with a port entry for every port, but don't enforce it with k8s policy management. This might still result in excessive load to the kube-apiserver (untested). This is not secure because the NetworkPolicy can be deleted or modified advertently or inadvertently, bypassing pod isolation.

Orchestrator(e.g. Kubernetes, Docker):
Kubernetes

Operating System (Linux/Windows):
Linux

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
What's noted in https://kubernetes.io/docs/concepts/services-networking/network-policies/ is the behavior we see when endPort is specified in a NetworkPolicy:

"Note: Your cluster must be using a CNI plugin that supports the endPort field in NetworkPolicy specifications. If your network plugin does not support the endPort field and you specify a NetworkPolicy with that, the policy will be applied only for the single port field."

The resulting NetworkPolicy resource applied only has the single port field applied instead of port and endPort.

[huntergregory]

Hello, both Linux and Windows NPM support port ranges (endPort).

The latest NPM versions are v1.4.45.2 (Linux) and v1.5.5 (Windows). If your AKS cluster has an older version of NPM, please contact Azure support and we will help you migrate.

We will close this issue, but if you are still encountering a problem with the latest version of Azure NPM, please (re)open an issue with your specifical yaml configuration.

Further Info

Automated Testing

We use Cyclonus and upstream e2e Conformance tests to verify NPM's behavior.

Manual Test

Below, we also include a manual validation of port ranges.

Pods

Make a duplicate Pod called b. Schedule the Pods on a node labeled with kubectl label node <name> vm=0.

apiVersion: v1
kind: Pod
metadata:
  labels:
    pod: a
  name: a
  namespace: default
spec:
  containers:
  - command:
    - /agnhost
    - serve-hostname
    - --tcp
    - --http=false
    - --port
    - "80"
    image: k8s.gcr.io/e2e-test-images/agnhost:2.33
    imagePullPolicy: IfNotPresent
    name: cont-80-tcp
    ports:
    - containerPort: 80
      name: serve-80-tcp
      protocol: TCP
  - command:
    - /agnhost
    - serve-hostname
    - --tcp
    - --http=false
    - --port
    - "81"
    image: k8s.gcr.io/e2e-test-images/agnhost:2.33
    imagePullPolicy: IfNotPresent
    name: cont-81-tcp
    ports:
    - containerPort: 81
      name: serve-81-tcp
      protocol: TCP
  nodeSelector:
    vm: "0"

NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: base
  namespace: default
spec:
  podSelector:
    matchLabels:
      pod: a
  policyTypes:
    - Egress
  egress:
    - to:
      ports:
        - protocol: TCP
          port: 80
          endPort: 81

Tests

Using:

kubectl exec -n default a -- /agnhost connect --timeout=3s $ipForPodB:80

[mitre-dleung]

Hello, both Linux and Windows NPM support port ranges (endPort).

The latest NPM versions are v1.4.45.2 (Linux) and v1.5.5 (Windows). If your AKS cluster has an older version of NPM, please contact Azure support and we will help you migrate.

We will close this issue, but if you are still encountering a problem with the latest version of Azure NPM, please (re)open an issue with your specifical yaml configuration.

Further Info

Automated Testing

We use Cyclonus and upstream e2e Conformance tests to verify NPM's behavior.

Manual Test

Below, we also include a manual validation of port ranges.

Pods

Make a duplicate Pod called b. Schedule the Pods on a node labeled with kubectl label node <name> vm=0.

apiVersion: v1
kind: Pod
metadata:
  labels:
    pod: a
  name: a
  namespace: default
spec:
  containers:
  - command:
    - /agnhost
    - serve-hostname
    - --tcp
    - --http=false
    - --port
    - "80"
    image: k8s.gcr.io/e2e-test-images/agnhost:2.33
    imagePullPolicy: IfNotPresent
    name: cont-80-tcp
    ports:
    - containerPort: 80
      name: serve-80-tcp
      protocol: TCP
  - command:
    - /agnhost
    - serve-hostname
    - --tcp
    - --http=false
    - --port
    - "81"
    image: k8s.gcr.io/e2e-test-images/agnhost:2.33
    imagePullPolicy: IfNotPresent
    name: cont-81-tcp
    ports:
    - containerPort: 81
      name: serve-81-tcp
      protocol: TCP
  nodeSelector:
    vm: "0"

NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: base
  namespace: default
spec:
  podSelector:
    matchLabels:
      pod: a
  policyTypes:
    - Egress
  egress:
    - to:
      ports:
        - protocol: TCP
          port: 80
          endPort: 81

Tests

Using:

kubectl exec -n default a -- /agnhost connect --timeout=3s $ipForPodB:80

Thanks for the details. I ran your test and confirmed I was able to reach the test service on port 80 and 81. This confirms the port range with endPort is working. This did however confirm that kubectl describe does not properly display the NetworkPolicy.

$ k describe -n default netpol base
Name:         base
Namespace:    default
Created on:   2023-10-25 15:19:00 +0000 UTC
Labels:       <none>
Annotations:  <none>
Spec:
  PodSelector:     pod=a
  Not affecting ingress traffic
  Allowing egress traffic:
    To Port: 80/TCP
    To: <any> (traffic not restricted by destination)
  Policy Types: Egress

Until kubectl describe is fixed, the workaround is to use the yaml output to view the NetworkPolicy.

$ k get -n default netpol base -o yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"NetworkPolicy","metadata":{"annotations":{},"name":"base","namespace":"default"},"spec":{"egress":[{"ports":[{"endPort":81,"port":80,"protocol":"TCP"}],"to":null}],"podSelector":{"matchLabels":{"pod":"a"}},"policyTypes":["Egress"]}}
  creationTimestamp: "2023-10-25T15:19:00Z"
  generation: 1
  name: base
  namespace: default
  resourceVersion: "2827142301"
  uid: 6e46a68a-1ca8-4be9-b6a7-689d6a400174
spec:
  egress:
  - ports:
    - endPort: 81
      port: 80
      protocol: TCP
  podSelector:
    matchLabels:
      pod: a
  policyTypes:
  - Egress
status: {}

[huntergregory]

Oh interesting, thanks for sharing. Created a new issue for this