Bug: nofile soft limit on EKS Fargate causes connection limits and crashes

[visit1985]

Summary

We have workloads running on EKS Fargate with an aws-appmesh-envoy sidecar injected by AWS App Mesh Controller.
The appnet agent process (PID 1) has a nofile soft limit of 65535, while the forked envoy process has a nofile soft limit of 1024 only.

kubectl exec -i -t -n default example-5ff7dbfc5d-strcr -c envoy -- sh
sh-4.2$ cat /proc/1/cmdline; echo
/usr/bin/agent
sh-4.2$ grep open /proc/1/limits
Max open files            65535                65535                files
sh-4.2$ cat /proc/31/cmdline; echo
/usr/bin/envoy-c/tmp/envoy-config-459706937.yaml-linfo--drain-time-s20
sh-4.2$ grep open /proc/31/limits
Max open files            1024                 65535                files

This imposes a limits of max. ~480 possible TCP connections, since a file handle is created for each ingress/egress.
Reaching the limit causes the envoy process to crash and being restarted by the appnet agent (#181), which causes outage.

Steps to Reproduce

Please refer to support case 170713370901828 for this.

Are you currently working around this issue?

We are unable to workaround this issue, because the appnet agent seems to be closed source.

[karanvasnani]

Thanks for your patience, continuing to track this investigation as part of aws/aws-app-mesh-roadmap#489

[karanvasnani]

Re-opening this issue since the fix hasn't been released yet. As an update, we experienced delays in our release and are currently working on a new release which will include this fix. Will share an update as soon as we have one.

[liubnu]

Close for aws/aws-app-mesh-roadmap#492