Increase nf_conntrack_tcp_timeout_close_wait

[grampelberg]

What problem are you trying to solve?

By default, nf_conntrack_tcp_timeout_close_wait is set to 60 seconds. Any requests taking longer than that get lost by the kernel causing 502 gateway errors to applications. kube-proxy bumped this up to 3600 seconds for its rules (see kubernetes/kubernetes#32551).

How should the problem be solved?

It is not possible to modify this via. the pod spec as it is marked "unsafe" in most k8s installations. However, as it is part of the network namespace, you can set it in init containers and have it persist for the life of the session. We should do this inside the proxy-init container and as part of setup for CNI.

[adleong]

In a quick spot check, I saw that this is already configured to 3600 on GKE and 60 on AKS.

[adleong]

It seems that in order to do this, we need to set allowPrivilegeEscalation and privilaged to true in the init container security policy. I'm not really sure what the implications of that are. Is this a tradeoff we're okay with?

[derrickburns]

I deployed this today into a test environment using an initContainer per the suggestion of @grampelberg. I will have feedback in the next few days if this worked to address the issues that we are seeing.

│   initContainers:                                                                                                     │
│   - args:                                                                                                             │
│     - -c                                                                                                              │
│     - sysctl -w net.netfilter.nf_conntrack_tcp_timeout_close_wait=3600                                                │
│     command:                                                                                                          │
│     - /bin/sh                                                                                                         │
│     image: busybox:1.29                                                                                               │
│     imagePullPolicy: IfNotPresent                                                                                     │
│     name: sysctl-buddy                                                                                                │
│     resources:                                                                                                        │
│       requests:                                                                                                       │
│         cpu: 1m                                                                                                       │
│         memory: 1Mi                                                                                                   │
│     securityContext:                                                                                                  │
│       privileged: true                                                                                                │
│     terminationMessagePath: /dev/termination-log                                                                      │
│     terminationMessagePolicy: File                                                                                    │
│     volumeMounts:                                                                                                     │
│     - mountPath: /var/run/secrets/kubernetes.io/serviceaccount                                                        │
│       name: gateway-proxy-token-28bpm                                                                                 │
│       readOnly: true

I shelled into a pod with this and did this:

/modsecurity/data # sysctl -a | grep  net.netfilter.nf_conntrack_tcp_timeout_close_wait
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 3600

[ericsuhong]

For us, we did not have to increase nf_conntrack_tcp_timeout_close_wait for all containers.

We only needed to increase nf_conntrack_tcp_timeout_close_wait value for Nginx ingress controller only to fix the issue, so maybe making this configurable as an annotation can limit the security concern.