Strange NetworkPolicy behavior with kube-network-policies

[gjabell]

This may be a misconfiguration or a bug, hence the discussion.

I am running the default Flannel CNI configuration on my Talos cluster, with the new kubeNetworkPoliciesEnabled option enabled. Most of the network policies work as expected, however it seems some traffic is being sporadically blocked without an obvious reason.

One case is connections to the Kubernetes API from application pods, for example cnpg databases:

I0519 12:58:17.955172       1 controller.go:260] "Processing sync for packet" id=6479
I0519 12:58:17.955216       1 logging.go:67] "Evaluating packet" direction="Egress" srcPod="external" dstPod="authelia/authelia-4" packet=<
        [6479] 192.168.47.12:6443 10.244.3.119:47542 TCP
 >
I0519 12:58:17.955243       1 networkpolicy.go:205] "Egress NetworkPolicies does not apply"
I0519 12:58:17.955256       1 logging.go:67] "Evaluating packet" direction="Ingress" srcPod="external" dstPod="authelia/authelia-4" packet=<
        [6479] 192.168.47.12:6443 10.244.3.119:47542 TCP
 >
I0519 12:58:17.955403       1 networkpolicy.go:334] "Pod is not allowed to be connected on port" pod="authelia/authelia-4" src-port=47542
I0519 12:58:17.955428       1 networkpolicy.go:334] "Pod is not allowed to be connected on port" pod="authelia/authelia-4" src-port=47542
I0519 12:58:17.955444       1 engine.go:67] "Packet denied by ingress policy"
I0519 12:58:17.955548       1 controller.go:276] "Finished syncing packet" id=6479 duration="378.112µs" verdict="drop"

No firewalls are configured on the host machines, and the failures here are not consistent (i.e. it sometimes works and sometimes doesn't). It looks like the network policy is blocking the established connection for some reason, although I'm not a network expert so maybe I'm misreading it.

Another issue I've noticed:

drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)
drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)
drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)
drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)
drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)
drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)
drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)
drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)
drop: I external (10.244.1.1:47868) -> victoria-logs/vlc-victoria-logs-cluster-vlselect-8649d7d469-rpb92 (10.244.1.184:9471) (TCP)

The formatting is different as I switched to JSON logging and wrote a script to match up the logs for easier inspection. It's not clear to me where these packets are coming from, as they are marked as "external" pods but with an IP address from within the cluster.

Has anyone run into similar issues with the new kube-network-policies integration?

[ahyanistheEmty]

kubeNetworkPoliciesEnabled only enforces Kubernetes‑style policies when the CNI itself supports them. The default Flannel CNI does not implement egress rules, so the policy engine falls back to the “drop if no matching egress rule” path. That’s why the API‑server connection (port 6443) is sometimes denied - the packet hits the policy check before the connection is fully established and, without an explicit egress allow rule, the engine drops it.

Fixes

1. Add a permissive egress rule that covers the API server (or all external destinations) for the pods that need it, e.g.:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-egress-to-apiserver
 namespace: authelia
spec:
 podSelector: {}
 policyTypes:
 - Egress
 egress:
 - to:
 - ipBlock:
 cidr: 192.168.47.0/24 # your control‑plane subnet
 ports:
 - protocol: TCP
 port: 6443

Adding a “allow all egress” policy (empty egress: list) is also a quick way to verify that the issue is policy‑related.

[gjabell]

Thanks for taking a look! I think this might be a different issue, as it's not the egress traffic which is blocked, rather the ingress (the logs mention "Egress NetworkPolicies does not apply" followed by "Packet denied by ingress policy"). I did try adding an allow-all egress policy but that does not make a difference as it's still only the ingress traffic which is being dropped.

I'm a bit confused at the claim that the kubeNetworkPoliciesEnabled only enforces when the CNI supports it, as this option is only available for Flannel, specifically because it does not have out-of-the-box support for network policies to begin with.

[ahyanistheEmty]

Ah yeah, you’re right, my bad, I was looking at it from the wrong angle initially.

Reading the logs again, this actually doesn’t look like an egress issue at all since it literally says:

"Egress NetworkPolicies does not apply"

and then immediately after it gets dropped by ingress policy instead.

So now it kinda looks like the initial connection is fine, but the return traffic from the API server is sometimes being treated as a completely new ingress connection instead of part of an already-established flow.

The thing that really makes me suspicious is the "srcPod="external"" part, especially in the second example where the source IP is still inside the cluster CIDR ("10.244.x.x"). That doesn’t seem right.

Since it’s intermittent too, I’m leaning more toward some bug/race condition in the new kube-network-policy integration rather than an actual bad policy config.

Could be:

• conntrack/state tracking acting up
• pod/IP resolution occasionally failing
• or maybe something Flannel-backend-specific

The "10.244.1.1 -> pod" drops especially feel like cluster traffic getting misclassified somehow.

[gjabell]

No worries!

My thoughts exactly; a race condition makes the most sense and since it's a new integration it seems likely that there might be some initial bugs. I'd be curious to see if anyone else enables it and runs into similar issues as it could still be a weird artifact from my network. I'm guessing that most people either run Flannel without the additional network policies or switch to a CNI which integrates them by default, so probably will have to wait for more data points to see if it's a general problem.

I checked the kube-network-policies repo as well but I suspect this is one of the first projects to actually integrate it as there's very few issues reported there so far.