-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Network policy in dual-stack cluster stops allowing ingress after pod restart #10053
Comments
This works fine for me. Are you perhaps just trying to make a request very quickly, before the network policies have synced? They are not instantaneous, there may be a few seconds following the creation of a new pod where the policies are not yet in effect. / # while true; do wget -qO - http://10.42.0.4; sleep 1; done
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.4
IP: fe80::48f4:edff:fe25:9ac5
RemoteAddr: 10.42.0.5:60158
GET / HTTP/1.1
Host: 10.42.0.4
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.4
IP: fe80::48f4:edff:fe25:9ac5
RemoteAddr: 10.42.0.5:60172
GET / HTTP/1.1
Host: 10.42.0.4
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.4
IP: fe80::48f4:edff:fe25:9ac5
RemoteAddr: 10.42.0.5:60174
GET / HTTP/1.1
Host: 10.42.0.4
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.4
IP: fe80::48f4:edff:fe25:9ac5
RemoteAddr: 10.42.0.5:49488
GET / HTTP/1.1
Host: 10.42.0.4
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.4
IP: fe80::48f4:edff:fe25:9ac5
RemoteAddr: 10.42.0.5:49490
GET / HTTP/1.1
Host: 10.42.0.4
User-Agent: Wget
Connection: close
# ---
# --- here is where I deleted the server pod - note that I get only a single error and then no further output until I hit control-c to break out of the loop
# ---
wget: can't connect to remote host (10.42.0.4): Connection refused
^C
# ---
# --- I then re-created the server pod and started querying it at its new address within a few seconds
# ---
/ # while true; do wget -qO - http://10.42.0.6; sleep 1; done
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.6
IP: fe80::200e:d6ff:fe7c:26d1
RemoteAddr: 10.42.0.5:43828
GET / HTTP/1.1
Host: 10.42.0.6
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.6
IP: fe80::200e:d6ff:fe7c:26d1
RemoteAddr: 10.42.0.5:43832
GET / HTTP/1.1
Host: 10.42.0.6
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.6
IP: fe80::200e:d6ff:fe7c:26d1
RemoteAddr: 10.42.0.5:43834
GET / HTTP/1.1
Host: 10.42.0.6
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.6
IP: fe80::200e:d6ff:fe7c:26d1
RemoteAddr: 10.42.0.5:43836
GET / HTTP/1.1
Host: 10.42.0.6
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.6
IP: fe80::200e:d6ff:fe7c:26d1
RemoteAddr: 10.42.0.5:43848
GET / HTTP/1.1
Host: 10.42.0.6
User-Agent: Wget
Connection: close
Hostname: server
IP: 127.0.0.1
IP: ::1
IP: 10.42.0.6
IP: fe80::200e:d6ff:fe7c:26d1
RemoteAddr: 10.42.0.5:43860
GET / HTTP/1.1
Host: 10.42.0.6
User-Agent: Wget
Connection: close |
That's odd. No, I've tried waiting for as long as several minutes and for me connectivity is never restored. I've created a new repo here with the manifests I'm using to test, plus a script that puts it all together to make sure we're doing the exact same things. You should be able to replicate by just spinning up a new server, then:
Note that I did add a service to front the I've been testing on VMs from Vultr, if that makes any difference. I can't think why it should but if it still doesn't replicate I don't know what else could be making the difference. |
I used the exact pod manifests you provided in your initial message. Is there anything else odd about your configuration? Are you sure it's network policy that is blocking your traffic? |
I think it's likely that the network policy is at least involved, given that the issue goes away immediately if I remove the network policy. I notice that the responses you are getting from your |
Do you have any errors in the k3s logs? I wonder if something is going on with the ipv6 rule sync? |
@manuelbuil do you have any ideas on what else to look at here? |
I have a similar issue as well. Sometimes, when I restart the pods, the network policy does not apply for minutes. Making changes to the network policy or deleting the pod again causes it to work sometimes as well. node-external-ip: "<ext4>,<ext6>"
flannel-iface: eno2
flannel-backend: host-gw
flannel-ipv6-masq: true
tls-san: <domain>
cluster-domain: <domain>
cluster-cidr: "<int4>/13,<int6>/56"
service-cidr: "<int4/16,<int6>/112"
secrets-encryption: true
kube-controller-manager-arg:
- "node-cidr-mask-size-ipv4=23"
- "node-cidr-mask-size-ipv6=64"
etcd-expose-metrics: true
disable: local-storage The problem becomes more frequent when using Unfortunately, I don't have much details yet but I'll try to do some investigation. |
Just tested this, going from a working state (pod running and receiving traffic) to a broken state (pod running but inaccessible) via delete / recreate. After deleting the working pod, I get this output:
When I then re-create the pod, I get this:
Finally, when I try (and fail) to make an http request to the running pod, sometimes I will see output like this:
However, that last doesn't happen all the time. In addition, separately from my running any tests, I have seen this message come up:
But I think that's just because I used a publicly-routable IPv6 range that doesn't actually route to the server I'm using. |
Thanks for reporting this, I was able to reproduce it. I suspect this is a bug in the kube-router network policy controller. When following your steps on a dual-stack environment, the first time I create the server, if I execute
After removing the server and recreating again, I can only see the ipv6 address:
The ipv4 memeber list has no IPs available. As a consequence, the verdict will always be false when the iptables rule checks that ipset. Workaround: Change your |
Hi @manuelbuil, thanks for the workaround! Can confirm it works in my testing environment. |
It's definitely a bug in upstream kube-router. The problem is that each ipFamily (ipv4 & ipv6) carries an ipset handler object. However, that ipset handler is tracking ipsets of both ipFamilies but only updates its ipFamily ipset correctly, the other one is left with outdated data (especially ipv6). As the ipv6 ipset is always the second one to be "refreshed", it overwrites the ipv4 ipset with that outdated data. I'm going to collect more information and open a bug issue in kube-router upstream |
kube-router maintainer created a fix. I tested and it fixes the issue :) |
Wow, that's great! Thanks for getting this addressed so quickly! |
Validated using commitid Thank you so much for this report and very clear reproduction steps! The PR linked has the steps and details of the test results for those curious. This fix will be available for general consumption in the June patch releases. |
Great, thanks so much for the speedy response! |
Environmental Info:
K3s Version: v1.29.4+k3s1 (94e29e2)
go version go1.21.9
Node(s) CPU architecture, OS, and Version:
Linux kube-test 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
single-node cluster for testing
Describe the bug:
When running a cluster in dual-stack mode, network policies that allow specific ingress to targeted pods stop allowing that ingress after the pod has been destroyed/re-created.
Steps To Reproduce:
server
pod from aboveExpected behavior:
Server should be reachable from client regardless of restarts.
Actual behavior:
Server is reachable from client only until server has been restarted, after which it goes dark.
Additional context / logs:
It's interesting to me that the connection is being refused, rather than timing out. I may be misremembering, but I thought that network policies simply dropped traffic that wasn't permitted, rather than refusing it. I could be misremembering, though.
Behavior initially observed on K3s v1.28.8+k3s1, upgraded to v1.29.4+k3s1 to see if that would fix it but no dice.
The text was updated successfully, but these errors were encountered: