If proxy port is taken on node, cilium will crashloop

[anfernee]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

1. It happens very rare in our test grid. When it happens we observe crashlooping cilium:

Name:                 cilium-fsqd5
Namespace:            kube-system

Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Normal   Pulled   25m (x65 over 5h26m)    kubelet  Container image "gcr.io/anthos-edgecontainer-release/cilium/cilium:v1.11.7-anthos1.14-gke3.0.3" already present on machine
  Warning  BackOff  58s (x1495 over 5h25m)  kubelet  Back-off restarting failed container

2. the cilium log says:

2022-11-16T01:35:04.841353158Z stderr F level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=10.4.0.0/16
2022-11-16T01:35:04.841362441Z stderr F level=info msg="Initializing daemon" subsys=daemon
2022-11-16T01:35:04.882162787Z stderr F level=info msg="Establishing connection to apiserver" host="https://35.202.63.6:6443" subsys=k8s
2022-11-16T01:35:04.934070731Z stderr F level=info msg="Connected to apiserver" subsys=k8s
2022-11-16T01:35:04.948171533Z stderr F level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"enable-host-reachable-services\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
2022-11-16T01:35:05.187120354Z stderr F level=info msg="Restored services from maps" failed=0 restored=0 subsys=service
2022-11-16T01:35:05.187311329Z stderr F level=info msg="Reading old endpoints..." subsys=daemon
2022-11-16T01:35:05.187345526Z stderr F level=info msg="No old endpoints found." subsys=daemon
2022-11-16T01:35:05.187423583Z stderr F level=info msg="Envoy: Starting xDS gRPC server listening on /var/run/cilium/xds.sock" subsys=envoy-manager
2022-11-16T01:35:05.188911845Z stderr F level=warning msg="Attempt to bind DNS Proxy failed, retrying in 4s" error="listen udp 0.0.0.0:40963: bind: address already in use" subsys=fqdn/dnsproxy
2022-11-16T01:35:09.189686261Z stderr F level=warning msg="Attempt to bind DNS Proxy failed, retrying in 4s" error="listen udp 0.0.0.0:40963: bind: address already in use" subsys=fqdn/dnsproxy
2022-11-16T01:35:13.190614861Z stderr F level=warning msg="Attempt to bind DNS Proxy failed, retrying in 4s" error="listen udp 0.0.0.0:40963: bind: address already in use" subsys=fqdn/dnsproxy
2022-11-16T01:35:17.191445821Z stderr F level=warning msg="Attempt to bind DNS Proxy failed, retrying in 4s" error="listen udp 0.0.0.0:40963: bind: address already in use" subsys=fqdn/dnsproxy
2022-11-16T01:35:21.192432893Z stderr F level=warning msg="Attempt to bind DNS Proxy failed, retrying in 4s" error="listen udp 0.0.0.0:40963: bind: address already in use" subsys=fqdn/dnsproxy
2022-11-16T01:35:25.192647526Z stderr F level=fatal msg="Error while creating daemon" error="listen udp 0.0.0.0:40963: bind: address already in use" subsys=daemon

3. Run sudo lsof -i -P -n to get port usage status in den1401 Spot that the port 40963 is used by command envelope_

COMMAND      PID            USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
envelope_   2483         prodbin   46u  IPv6   76910      0t0  UDP *:40963

4. Looking into the code, cilium does have port allocation logic to find free port to use. However, it also tries to recover port from existing iptables rules, which shadow the port allocation logic.

cilium/pkg/datapath/iptables/iptables.go

Lines 936 to 956 in 3a15990

	func (m *IptablesManager) doGetProxyPort(prog iptablesInterface, name string) uint16 {
	rules, err := prog.runProgOutput([]string{"-t", "mangle", "-n", "-L", ciliumPreMangleChain})
	if err != nil {
	return 0
	}
	re := regexp.MustCompile(name + ".*TPROXY redirect (0.0.0.0|::):([1-9][0-9]*) mark")
	strs := re.FindAllString(rules, -1)
	if len(strs) == 0 {
	return 0
	}
	// Pick the port number from the last match, as rules are appended to the end (-A)
	portStr := re.ReplaceAllString(strs[len(strs)-1], "$2")
	portUInt64, err := strconv.ParseUint(portStr, 10, 16)
	if err != nil {
	log.WithError(err).Debugf("Port number cannot be parsed: %s", portStr)
	return 0
	}
	return uint16(portUInt64)
	}

Cilium Version

Discovered in v1.11. Suspect it also exists on master.

Kernel Version

not kernel specific.

Kubernetes Version

not k8s specific

Sysdump

No response

Relevant log output

No response

Anything else?

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[github-actions]

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

[github-actions]

This issue has not seen any activity since it was marked stale.
Closing.