Internal networking does not work with encryption enabled until a rollout restart is performed #19958

msarti · 2022-05-25T12:19:25Z

Is there an existing issue for this?

I have searched the existing issues

What happened?

After installation of Cilium on a newly bootstrapped cluster with wireguard encryption enabled, the pods do not reach any addresses within the cluster. Issue does not appear if I do not enable encryption.
It is possible to get around the issue with a rollout restart of the cilium daemonset.

values.yaml:

kubeProxyReplacement: strict
l7Proxy: false
encryption:
  enabled: true
  type: wireguard
ipam:
  mode: "cluster-pool"
  operator:
    clusterPoolIPv4PodCIDRList: "10.244.0.0/16"
    clusterPoolIPv4MaskSize: "24"
hostServices:
  enabled: true
k8sServiceHost: 192.168.0.100
k8sServicePort: 8443

To reproduce the issue:

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

kubectl exec -i -t dnsutils -- nslookup kubernetes.default
;; connection timed out; no servers could be reached

command terminated with exit code 1

To work around the issue:

kubectl -n kube-system rollout restart ds/cilium

kubectl exec -i -t dnsutils -- nslookup kubernetes.default
Server:		10.97.0.10
Address:	10.97.0.10#53

Name:	kubernetes.default.svc.cluster.local
Address: 10.97.0.1

Cilium Version

Client: 1.11.5 b0d3140 2022-05-10T02:47:42+02:00 go version go1.17.9 linux/amd64
Daemon: 1.11.5 b0d3140 2022-05-10T02:47:42+02:00 go version go1.17.9 linux/amd64

Kernel Version

Linux default-cp-01 5.15.0-30-generic #31-Ubuntu SMP Thu May 5 10:00:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:46:05Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.0", GitCommit:"4ce5a8954017644c5420bae81d72b09b735c21f0", GitTreeState:"clean", BuildDate:"2022-05-03T13:38:19Z", GoVersion:"go1.18.1", Compiler:"gc", Platform:"linux/amd64"}

Sysdump

No response

Relevant log output

No relevant logs detected.

Anything else?

No response

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

kvaster · 2022-05-25T15:03:19Z

I'm investigating this problem already for 5 days. Problem is not only with encryption - it is just really really visible when encryption is enabled.

I have test cluster with 4 nodes. All this nodes run in VM. Before starting test I'm launching cpu intensive task on host (i.e. compiling clang). After that I'm restarting all 4 nodes. And sometimes endpoint delete event is received after endpoint setup.
I've applied following patch for debugging:

test-patch.txt

And also following script to check endpoints:

check-cilium-node.rb.txt

After last restart I was able to see:

processing k8s-1 = 10.117.12.20
processing k8s-2 = 10.117.12.21
pod ip is unreachable: 10.244.0.174
processing k8s-3 = 10.117.12.22
pod ip is unreachable: 10.244.0.174
processing k8s-4 = 10.117.12.23
pod ip is unreachable: 10.244.0.174

That means wg show on node (k8s-2...) will not have 10.244.0.174 in allowed ip list. Also cilium endpoint list will not show this IP.

From the log (only lines with this IP):

level=info msg="!!! no peer updated" ipAddr=10.244.0.174/32 modification=Upsert newNode=10.117.12.20 oldNode="<nil>" subsys=wireguard
level=info msg="!!! node lookup failed on upsert" oldHostIP=10.117.12.20 subsys=wireguard
level=info msg="!!! peer update - allowed ips" allowedIPs="[{10.244.0.40 ffffffff} {10.244.0.131 ffffffff} {10.244.0.102 ffffffff} {10.244.0.111 ffffffff} {10.244.0.99 ffffffff} {10.244.0.142 ffffffff} {10.244.0.53 ffffffff} {10.244.0.247 ffffffff} {10.244.0.112 ffffffff} {10.244.0.174 ffffffff} {10.244.0.47 ffffffff} {10.244.0.34 ffffffff} {10.244.0.54 ffffffff} {10.244.0.101 ffffffff}]" subsys=wireguard
level=info msg="!!! Updating peer config" endpoint="10.117.12.20:51871" ipAddrs="[{10.244.0.40 ffffffff} {10.244.0.131 ffffffff} {10.244.0.102 ffffffff} {10.244.0.111 ffffffff} {10.244.0.99 ffffffff} {10.244.0.142 ffffffff} {10.244.0.53 ffffffff} {10.244.0.247 ffffffff} {10.244.0.112 ffffffff} {10.244.0.174 ffffffff} {10.244.0.47 ffffffff} {10.244.0.34 ffffffff} {10.244.0.54 ffffffff} {10.244.0.101 ffffffff}]" pubKey="+Fr/wILB8LakWutxz96o15BM38UosPqP4iSX+ehYnVM=" subsys=wireguard
...
level=info msg="!!! insert ip - already exists" ip=10.244.0.174/32 subsys=wireguard
level=info msg="!!! insert peer failed" ipnet="{10.244.0.174 ffffffff}" nodeName=k8s-1 subsys=wireguard
level=info msg="!!! no peer updated" ipAddr=10.244.0.174/32 modification=Upsert newNode=10.117.12.20 oldNode=10.117.12.20 subsys=wireguard
level=info msg="!!! Updating peer config" endpoint="10.117.12.20:51871" ipAddrs="[{10.244.0.40 ffffffff} {10.244.0.131 ffffffff} {10.244.0.102 ffffffff} {10.244.0.99 ffffffff} {10.244.0.142 ffffffff} {10.244.0.53 ffffffff} {10.244.0.247 ffffffff} {10.244.0.174 ffffffff} {10.244.0.34 ffffffff} {10.244.0.54 ffffffff} {10.244.0.101 ffffffff} {10.244.0.223 ffffffff} {10.244.0.98 ffffffff}]" pubKey="+Fr/wILB8LakWutxz96o15BM38UosPqP4iSX+ehYnVM=" subsys=wireguard
...
level=info msg="!!! Updating peer config" endpoint="10.117.12.20:51871" ipAddrs="[{10.244.0.40 ffffffff} {10.244.0.131 ffffffff} {10.244.0.102 ffffffff} {10.244.0.99 ffffffff} {10.244.0.142 ffffffff} {10.244.0.53 ffffffff} {10.244.0.247 ffffffff} {10.244.0.174 ffffffff} {10.244.0.34 ffffffff} {10.244.0.54 ffffffff} {10.244.0.101 ffffffff} {10.244.0.223 ffffffff} {10.244.0.98 ffffffff} {10.244.0.222 ffffffff}]" pubKey="+Fr/wILB8LakWutxz96o15BM38UosPqP4iSX+ehYnVM=" subsys=wireguard
level=info msg="!!! peer updated" ipAddr=10.244.0.222/32 modification=Upsert newNode=10.117.12.20 oldNode="<nil>" pubKey="+Fr/wILB8LakWutxz96o15BM38UosPqP4iSX+ehYnVM=" subsys=wireguard
level=info msg="-!- Delete ip on metadata match" ipAddr=10.244.0.174 subsys=ipcache
level=info msg="!!! Updating peer config" endpoint="10.117.12.20:51871" ipAddrs="[{10.244.0.40 ffffffff} {10.244.0.131 ffffffff} {10.244.0.102 ffffffff} {10.244.0.99 ffffffff} {10.244.0.142 ffffffff} {10.244.0.53 ffffffff} {10.244.0.247 ffffffff} {10.244.0.34 ffffffff} {10.244.0.54 ffffffff} {10.244.0.101 ffffffff} {10.244.0.223 ffffffff} {10.244.0.98 ffffffff} {10.244.0.222 ffffffff}]" pubKey="+Fr/wILB8LakWutxz96o15BM38UosPqP4iSX+ehYnVM=" subsys=wireguard
level=info msg="!!! peer updated" ipAddr=10.244.0.174/32 modification=Delete newNode="<nil>" oldNode=10.117.12.20 pubKey="+Fr/wILB8LakWutxz96o15BM38UosPqP4iSX+ehYnVM=" subsys=wireguard
level=info msg="-!- Delete ip on metadata match" ipAddr= subsys=ipcache

kvaster · 2022-05-26T13:22:38Z

It seems that my problem is related to NodeGC - disabling node gc fixes the problem.

Also I had non-default params for k8s controller:

node-monitor-grace-period: 16s
node-monitor-period: 2s
pod-eviction-timeout: 30s

I decided to make some experiments and removed this params (without disabling NodeGC in cilium). Problem was gone in 90% cases. After that I've applied following params:

node-startup-grace-period: 5m0s
node-monitor-grace-period: 2m0s

And there is no more problem with such config...

kvaster · 2022-05-28T14:13:39Z

I was wrong. Problem is still reproduceable with ANY timeout and with disabled ciliume endpoint gc, cilium identiy gc e.t.c. It's enough to restart only one node...

Node restart may lead to ip addresses 'swap' for services. In such case using ipcache.Delete may lead to deleting newly assigned ip address of other service. Fixes: cilium#19958 Signed-off-by: Viktor Kuzmin <kvaster@gmail.com>

Node restart may lead to ip addresses 'swap' for services. In such case using ipcache.Delete may lead to deleting newly assigned ip address of other service. Fixes: #19958 Signed-off-by: Viktor Kuzmin <kvaster@gmail.com>

[ upstream commit 4bf70f1 ] [ Backporterter's note: There was a conflict due to cilium@a445f69 from cilium#19073 not have been backported. I opted for modidyfing the patch to call into the global IPCAche ] Node restart may lead to ip addresses 'swap' for services. In such case using ipcache.Delete may lead to deleting newly assigned ip address of other service. Fixes: cilium#19958 Signed-off-by: Viktor Kuzmin <kvaster@gmail.com> Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>

[ upstream commit 4bf70f1 ] [ Backporterter's note: There was a conflict due to a445f69 from #19073 not have been backported. I opted for modidyfing the patch to call into the global IPCAche ] Node restart may lead to ip addresses 'swap' for services. In such case using ipcache.Delete may lead to deleting newly assigned ip address of other service. Fixes: #19958 Signed-off-by: Viktor Kuzmin <kvaster@gmail.com> Signed-off-by: Kornilios Kourtis <kornilios@isovalent.com>

msarti added kind/bug This is a bug in the Cilium logic. needs/triage This issue requires triaging to establish severity and next steps. labels May 25, 2022

kvaster mentioned this issue May 29, 2022

Use DeleteOnMetadataMatch instead of Delete for endpointUpdated #19996

Merged

pchaigno closed this as completed in #19996 Jun 15, 2022

kkourt removed the needs/triage This issue requires triaging to establish severity and next steps. label Jun 21, 2022

alphanoob1337 mentioned this issue Jan 9, 2023

Connectivity test failing at lookup for kubernetes.default after fresh install #22997

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Internal networking does not work with encryption enabled until a rollout restart is performed #19958

Internal networking does not work with encryption enabled until a rollout restart is performed #19958

msarti commented May 25, 2022

kvaster commented May 25, 2022

kvaster commented May 26, 2022

kvaster commented May 28, 2022

Internal networking does not work with encryption enabled until a rollout restart is performed #19958

Internal networking does not work with encryption enabled until a rollout restart is performed #19958

Comments

msarti commented May 25, 2022

Is there an existing issue for this?

What happened?

Cilium Version

Kernel Version

Kubernetes Version

Sysdump

Relevant log output

Anything else?

Code of Conduct

kvaster commented May 25, 2022

kvaster commented May 26, 2022

kvaster commented May 28, 2022