Skip to content

enabling CONFIG_INET_DIAG_DESTROY (for Cilium in kube-proxy-replacement mode) #14108

Open
@joelddiaz

Description

@joelddiaz

Is your feature request related to a problem? Please describe.

When using Azure Linux nodes in a kubernetes AKS cluster with Cilium configured to run in kube-proxy-replacement mode, the underlying kernel is missing a configuration that allows Cilium to tear down stale networking connections to stale Pods (Pods that have been deleted due to rolling upgrades or scaling down events).

For TCP connections, this isn't a huge problem as most applications should notice the TCP connection is down, which should trigger a reconnection through the kubernetes Service to a new (running) Pod.

For UDP connections, this can leave an app attempting to send traffic through a socket connection that is no longer listening-for/receiving traffic.

Cilium requires the following kernel configuration settings to be able to destroy these connections to non-existent Pods: CONFIG_INET_DIAG, CONFIG_INET_UDP_DIAG, and CONFIG_INET_DIAG_DESTROY. See the section talking about CONFIG_INET_* here.

Here we see what is and isn't enabled on the underlying Azure Linux nodes:

aks-azsystem-15662614-vmss000000 $ cat /host/boot/config-6.6.82.1-1.azl3|  grep -e CONFIG_INET_DIA  -e CONFIG_INET_UDP
CONFIG_INET_DIAG=m
CONFIG_INET_UDP_DIAG=m
# CONFIG_INET_DIAG_DESTROY is not set

aks-azsystem-15662614-vmss000000 $ uname -a
Linux aks-azsystem-15662614-vmss000000 6.6.82.1-1.azl3 #1 SMP PREEMPT_DYNAMIC Wed Apr  2 02:28:07 UTC 2025 x86_64 Linux

Since the Azure Linux kernel is not enabled with CONFIG_INET_DIAG_DESTROY this keeps Cilium from being able to proactively tear down these stale UDP connections. In practice, we've seen apps like nginx maintaining UDP DNS connections to the stale Pod IPs causing DNS resolution errors.

Describe the solution you'd like

Enable CONFIG_INET_DIAG_DESTROY (as CONFIG_INET_DIAG and CONFIG_INET_UDP_DIAG are already enabled) so that Azure Linux can be successfully used as a base for kubernetes nodes when Cilium is configured in kube-proxy-replacement mode.

Describe alternatives you've considered

While investigating this, we noticed that using Ubuntu nodes for the AKS clusters do not suffer from this issue as the Ubuntu kubernetes node images contain a kernel with the necessary pieces enabled.

Here we see that the Ubuntu kernel has all the pieces necessary for Cilium to work properly:

aks-ubuntupool-80544669-vmss000000 $  cat /host/boot/config-5.15.0-1090-azure | grep -i -e CONFIG_INET_DIA -e CONFIG_INET_U
CONFIG_INET_DIAG=m
CONFIG_INET_UDP_DIAG=m
CONFIG_INET_DIAG_DESTROY=y

aks-ubuntupool-80544669-vmss000000:$ uname -a
Linux aks-ubuntupool-80544669-vmss000000 5.15.0-1090-azure #99-Ubuntu SMP Thu May 22 21:15:50 UTC 2025 x86_64 Linux

Since we're already using Azure Linux, we'd like to be able to continue to use it for our kubernetes AKS clusters rather than migrating to Ubuntu or mixing/matching between Azure Linux and Ubuntu nodes depending on the workloads that will run on the nodes.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature-requestRequest for a feature or enhancement

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions