-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Internal networking does not work with encryption enabled until a rollout restart is performed #19958
Comments
I'm investigating this problem already for 5 days. Problem is not only with encryption - it is just really really visible when encryption is enabled. I have test cluster with 4 nodes. All this nodes run in VM. Before starting test I'm launching cpu intensive task on host (i.e. compiling clang). After that I'm restarting all 4 nodes. And sometimes endpoint delete event is received after endpoint setup. And also following script to check endpoints: After last restart I was able to see:
That means From the log (only lines with this IP):
|
It seems that my problem is related to NodeGC - disabling node gc fixes the problem. Also I had non-default params for k8s controller:
I decided to make some experiments and removed this params (without disabling NodeGC in cilium). Problem was gone in 90% cases. After that I've applied following params:
And there is no more problem with such config... |
I was wrong. Problem is still reproduceable with ANY timeout and with disabled ciliume endpoint gc, cilium identiy gc e.t.c. It's enough to restart only one node... |
Node restart may lead to ip addresses 'swap' for services. In such case using ipcache.Delete may lead to deleting newly assigned ip address of other service. Fixes: cilium#19958 Signed-off-by: Viktor Kuzmin <kvaster@gmail.com>
Node restart may lead to ip addresses 'swap' for services. In such case using ipcache.Delete may lead to deleting newly assigned ip address of other service. Fixes: #19958 Signed-off-by: Viktor Kuzmin <kvaster@gmail.com>
[ upstream commit 4bf70f1 ] [ Backporterter's note: There was a conflict due to cilium@a445f69 from cilium#19073 not have been backported. I opted for modidyfing the patch to call into the global IPCAche ] Node restart may lead to ip addresses 'swap' for services. In such case using ipcache.Delete may lead to deleting newly assigned ip address of other service. Fixes: cilium#19958 Signed-off-by: Viktor Kuzmin <kvaster@gmail.com> Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
[ upstream commit 4bf70f1 ] [ Backporterter's note: There was a conflict due to a445f69 from #19073 not have been backported. I opted for modidyfing the patch to call into the global IPCAche ] Node restart may lead to ip addresses 'swap' for services. In such case using ipcache.Delete may lead to deleting newly assigned ip address of other service. Fixes: #19958 Signed-off-by: Viktor Kuzmin <kvaster@gmail.com> Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>
Is there an existing issue for this?
What happened?
After installation of Cilium on a newly bootstrapped cluster with wireguard encryption enabled, the pods do not reach any addresses within the cluster. Issue does not appear if I do not enable encryption.
It is possible to get around the issue with a rollout restart of the cilium daemonset.
values.yaml:
To reproduce the issue:
To work around the issue:
Cilium Version
Kernel Version
Linux default-cp-01 5.15.0-30-generic #31-Ubuntu SMP Thu May 5 10:00:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Kubernetes Version
Sysdump
No response
Relevant log output
Anything else?
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: