Change to network policy behavior after version 1.2.3

[stefansedich]

What happened:
We utilize https://zero-to-jupyterhub.readthedocs.io/en/latest/ within our environments which manages a network policy for the Jupyterhub hub component, this has been operating without issue for a number of agent releases until we attempted to upgrade the agent from 1.2.3 to 1.2.7.

After this upgrade we began to experience connective issues from the hub component to the API server, after some investigation we have discovered that there has been a change in behavior since version 1.2.3 that changes how policies are handled.

I have created a minimal reproduction based on the Jupyter network policy that demonstrates this issue, the following will work without issue running version 1.2.3 of the agent and allow egress from the pod to the API server, however when running version 1.2.7 of the agent the egress will be blocked.

Attach logs

Can provide logs if they will be beneficial here however the reproduction above clearly demonstrates the change in behavior.

What you expected to happen:

The egress to the API server from the pod will not be blocked.

How to reproduce it (as minimally and precisely as possible):

#!/bin/bash

echo "Creating pod..."
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: demo
  labels:
    app: demo
spec:
  containers:
  - image: curlimages/curl:latest
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: curl
  restartPolicy: Always
EOF

echo "Waiting for pod to be ready..."
kubectl wait --for=condition=ready pod/demo --timeout=300s

echo "Pod is ready! Running curl command..."
kubectl exec demo -- curl -v --connect-timeout 3 https://172.20.0.1

echo "Applying network policy..."
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test
spec:
  podSelector:
    matchLabels:
      app: demo
  egress:
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
    to:
    - ipBlock:
        cidr: 172.16.0.0/12
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 172.16.0.0/12
  - to:
    - ipBlock:
        cidr: 172.16.0.0/12
  policyTypes:
  - Egress
EOF

echo "Running curl command again after applying network policy..."
kubectl exec demo -- curl -v --connect-timeout 3 https://172.20.0.1

echo "Cleaning up - removing pod and network policy ..."
kubectl delete pod demo --force --grace-period=0
kubectl delete networkpolicy test

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.33.4
• CNI Version: 1.20.4
• Network Policy Agent Version: 1.2.7
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):

[viveksb007]

Hey @stefansedich
what is the intent with the policy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test
spec:
  podSelector:
    matchLabels:
      app: demo
  egress:
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
    to:
    - ipBlock:
        cidr: 172.16.0.0/12
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 172.16.0.0/12
  - to:
    - ipBlock:
        cidr: 172.16.0.0/12
  policyTypes:
  - Egress

with above policy its not clear what we are trying to do with CIDR 172.16.0.0/12

---

  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
    to:
    - ipBlock:
        cidr: 172.16.0.0/12

this above section is trying to allow port 53 TCP/UDP to CIDR 172.16.0.0/12

  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 172.16.0.0/12

this above section is trying to allow everything but the CIDR 172.16.0.0/12

  - to:
    - ipBlock:
        cidr: 172.16.0.0/12

this above section is trying to allow everything to CIDR 172.16.0.0/12

[stefansedich]

@viveksb007 as mentioned this was a simple reproduction of the issue we are seeing with the full policy for the hub component, here is the full policy that is created by the chart we are using, reference here https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/templates/hub/netpol.yaml

From what I understand they are doing it in this way so specific egress can be turned on/off with flags while maintaining a default deny to those ranges.

Regardless of this based on the k8s documentation of network policies it was my understanding that evaluation should be an OR and this was the case with version 1.2.3 of the agent and with Calico when we were using that as our policy engine however this behavior has changed in later versions of this agent causing policies that were previously working to no longer work.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: hub
spec:
  egress:
  - ports:
    - port: 8001
      protocol: TCP
    to:
    - podSelector:
        matchLabels:
          app: jupyterhub
          component: proxy
          release: engineering-jupyterhub-hub
  - ports:
    - port: 8888
      protocol: TCP
    to:
    - podSelector:
        matchLabels:
          app: jupyterhub
          component: singleuser-server
          release: engineering-jupyterhub-hub
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
    to:
    - ipBlock:
        cidr: 169.254.169.254/32
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    - ipBlock:
        cidr: 10.0.0.0/8
    - ipBlock:
        cidr: 172.16.0.0/12
    - ipBlock:
        cidr: 192.168.0.0/16
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - 169.254.169.254/32
  - to:
    - ipBlock:
        cidr: 10.0.0.0/8
    - ipBlock:
        cidr: 172.16.0.0/12
    - ipBlock:
        cidr: 192.168.0.0/16
  - to:
    - ipBlock:
        cidr: 169.254.169.254/32
  ingress:
  - from:
    - podSelector:
        matchLabels:
          hub.jupyter.org/network-access-hub: "true"
    ports:
    - port: http
      protocol: TCP
  podSelector:
    matchLabels:
      app: jupyterhub
      component: hub
      release: engineering-jupyterhub-hub
  policyTypes:
  - Ingress
  - Egress

[stefansedich]

@Pavani-Panakanti is the change in behavior perhaps related to the changes in #344 ?

[stefansedich]

So debugging this a little further it looks like the trie map is as I would expect given my original example:

Computing trie key: {10.1.1.1 ffffffff}
L4 values: protocol: 254 startPort: 0 endPort: 0
Parsed Except CIDR: 172.16.0.0/12

Updating Map with IP Key: 0.0.0.0/0
Computing trie key: {0.0.0.0 00000000}
L4 values: protocol: 254 startPort: 0 endPort: 0

Updating Map with IP Key: 172.16.0.0/12
Computing trie key: {172.16.0.0 fff00000}
L4 values: protocol: 254 startPort: 0 endPort: 0
L4 values: protocol: 17 startPort: 53 endPort: 0
L4 values: protocol: 6 startPort: 53 endPort: 0

However digging a little deeper and looking at the policy endpoint the controller creates, it appears to be missing the ANY rule for 172.16.0.0/12 entirely that exists in the network policy.

apiVersion: networking.k8s.aws/v1alpha1
kind: PolicyEndpoint
metadata:
    name: test-csl5v
spec:
  egress:
  - cidr: 172.16.0.0/12
    ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
  - cidr: 0.0.0.0/0
    except:
    - 172.16.0.0/12
  podIsolation:
  - Egress
  podSelector:
    matchLabels:
      app: demo
  policyRef:
    name: test

So this now explains to me why #344 changed the behavior we are seeing, and leads me to think the issue could live within the policy controller itself.

Will close this issue and open an issue over in the controller repo.