`encryption.wireguard.persistentKeepalive` is not working effectively

[fujitatomoya]

Is there an existing issue for this?

• I have searched the existing issues

Version

equal or higher than v1.16.0 and lower than v1.17.0

What happened?

I am not sure if this is a bug for cilium, but would like to learn the possibility or feedback from the community.

install cilium with cilium install --version v1.16.2 --set encryption.enabled=true,encryption.type=wireguard,encryption.wireguard.persistentKeepalive=1s, and check the persistentKeepalive package every 1 second iteration via tcpdump command as following.

tcpdump -pvn -i cilium_wg0 -tt
tcpdump: listening on cilium_wg0, link-type RAW (Raw IP), snapshot length 262144 bytes
1727905297.665510 IP (tos 0x0, ttl 64, id 51437, offset 0, flags [none], proto UDP (17), length 94)
    172.18.0.3.48703 > 172.18.0.2.8472: OTV, flags [I] (0x08), overlay 0, instance 6
IP (tos 0x0, ttl 64, id 12044, offset 0, flags [DF], proto ICMP (1), length 44)
    10.244.2.203 > 10.244.0.147: ICMP echo request, id 38309, seq 12743, length 24
1727905297.665768 IP (tos 0x0, ttl 64, id 25228, offset 0, flags [none], proto UDP (17), length 94)
    172.18.0.3.55144 > 172.18.0.4.8472: OTV, flags [I] (0x08), overlay 0, instance 6
IP (tos 0x0, ttl 64, id 36536, offset 0, flags [DF], proto ICMP (1), length 44)
    10.244.2.203 > 10.244.1.36: ICMP echo request, id 38309, seq 12743, length 24
1727905297.665810 IP (tos 0x0, ttl 64, id 1454, offset 0, flags [DF], proto ICMP (1), length 44)
    172.18.0.3 > 172.18.0.4: ICMP echo request, id 38309, seq 12743, length 24
1727905297.666034 IP (tos 0x0, ttl 64, id 9719, offset 0, flags [none], proto UDP (17), length 94)
    172.18.0.2.48703 > 172.18.0.3.8472: OTV, flags [I] (0x08), overlay 0, instance 4
IP (tos 0x0, ttl 64, id 56297, offset 0, flags [none], proto ICMP (1), length 44)
    10.244.0.147 > 10.244.2.203: ICMP echo reply, id 38309, seq 12743, length 24
...

i cannot confirm the keepalive packet 1 second iteration.

but it looks like wireguard configuration on the host is successfully set,

root@kind-worker:/home/cilium# wg
interface: cilium_wg0
  public key: aeT2hfuLPs3R8oLNtde/zB9hHvhAihm5fpwrKqLJeFc=
  private key: (hidden)
  listening port: 51871
  fwmark: 0x1e00

peer: e390Llh/aBVI0gkfFbvBCBx2rhPMSqdWjJFrXPWifls=
  endpoint: 172.18.0.2:51871
  allowed ips: 172.18.0.2/32, 10.244.0.194/32, 10.244.0.147/32, 10.244.0.26/32, 10.244.0.236/32, 10.244.0.123/32
  latest handshake: 56 seconds ago
  transfer: 385.13 KiB received, 388.71 KiB sent
  persistent keepalive: every 1 second

peer: O4ec+qOC8yCowQr8GqmrgCCoBHssxoUuFFohfhkjk0Y=
  endpoint: 172.18.0.4:51871
  allowed ips: 172.18.0.4/32, 10.244.1.157/32, 10.244.1.36/32
  latest handshake: 1 minute, 5 seconds ago
  transfer: 464.29 KiB received, 456.96 KiB sent
  persistent keepalive: every 1 second

How can we reproduce the issue?

1. install cilium with,

cilium install --version v1.16.2 --set encryption.enabled=true,encryption.type=wireguard,encryption.wireguard.persistentKeepalive=1s

2. use tcpdump to check the keepalive packet for each 1 sec iteration.

tcpdump -pvn -i cilium_wg0 -tt

Cilium Version

tomoyafujita@~/kind >cilium version
cilium-cli: v0.16.0-6-g70bd400f compiled with go1.22.1 on linux/amd64
cilium image (default): v1.15.1
cilium image (stable): v1.16.2
cilium image (running): 1.16.2

Kernel Version

tomoyafujita@~/kind >uname -a
Linux tomoyafujita 5.15.0-121-generic #131~20.04.1-Ubuntu SMP Mon Aug 12 13:09:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

tomoyafujita@~/kind >kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.15", GitCommit:"1649f592f1909b97aa3c2a0a8f968a3fd05a7b8b", GitTreeState:"clean", BuildDate:"2024-03-14T01:05:39Z", GoVersion:"go1.21.8", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.1", GitCommit:"4c9411232e10168d7b050c49a1b59f6df9d7ea4b", GitTreeState:"clean", BuildDate:"2023-05-12T19:03:40Z", GoVersion:"go1.20.3", Compiler:"gc", Platform:"linux/amd64"}

Regression

No response

Sysdump

cilium-sysdump-20241002-162642.zip

Relevant log output

No response

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[fujitatomoya]

Sorry i got confused with cilium_wg0 with Host NIC. I should have checked on host network interface, since the punch-hole and keepalive packets are incoming host network.

now i confirmed the keepalive packets are coming as expected frequency.

root@kind-control-plane:/home/cilium# tcpdump -n udp port 51871 -i eth0 -tt -pv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
1728509285.321988 IP (tos 0x0, ttl 64, id 3588, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.3.51871 > 172.18.0.4.51871: UDP, length 32
1728509285.578054 IP (tos 0x0, ttl 64, id 62348, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.2.51871 > 172.18.0.4.51871: UDP, length 32
1728509286.345863 IP (tos 0x0, ttl 64, id 5954, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.4.51871 > 172.18.0.3.51871: UDP, length 32
1728509286.345891 IP (tos 0x0, ttl 64, id 3684, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.3.51871 > 172.18.0.4.51871: UDP, length 32
1728509286.601926 IP (tos 0x0, ttl 64, id 27776, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.4.51871 > 172.18.0.2.51871: UDP, length 32
1728509286.601947 IP (tos 0x0, ttl 64, id 62579, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.2.51871 > 172.18.0.4.51871: UDP, length 32
1728509287.369936 IP (tos 0x0, ttl 64, id 6096, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.4.51871 > 172.18.0.3.51871: UDP, length 32
1728509287.625959 IP (tos 0x0, ttl 64, id 62648, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.2.51871 > 172.18.0.4.51871: UDP, length 32
1728509288.397997 IP (tos 0x0, ttl 64, id 6267, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.4.51871 > 172.18.0.3.51871: UDP, length 32
1728509288.398010 IP (tos 0x0, ttl 64, id 3829, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.3.51871 > 172.18.0.4.51871: UDP, length 32
1728509288.653855 IP (tos 0x0, ttl 64, id 27885, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.4.51871 > 172.18.0.2.51871: UDP, length 32
1728509288.653967 IP (tos 0x0, ttl 64, id 62650, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.2.51871 > 172.18.0.4.51871: UDP, length 32
1728509289.417980 IP (tos 0x0, ttl 64, id 6424, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.4.51871 > 172.18.0.3.51871: UDP, length 32
1728509289.418018 IP (tos 0x0, ttl 64, id 3967, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.3.51871 > 172.18.0.4.51871: UDP, length 32
1728509289.677870 IP (tos 0x0, ttl 64, id 28036, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.4.51871 > 172.18.0.2.51871: UDP, length 32
1728509289.677886 IP (tos 0x0, ttl 64, id 62673, offset 0, flags [none], proto UDP (17), length 60)
    172.18.0.2.51871 > 172.18.0.4.51871: UDP, length 32