Allow REUSEPORT for listener sockets

[mattklein123]

We should optionally allow listener sockets to be created on each worker with SO_REUSEPORT. This should probably be the default for UDP listeners and opt-in for TCP listeners?

cc @danzh2010 @euroelessar @antoniovicente

[danzh2010]

Hi Matt, actually this is part of the plan in https://docs.google.com/document/d/1My-Tssc9d6rW-Xu96y4DdUxvzIKmptP6mm6i_zOXr58/edit?pli=1#. I was about to modify ListenerConfig to provide a listen socket factory interface instead of a socket as the first step of preparation for supporting multiple quic listeners on different worker threads.

What's your plan for this?

[mattklein123]

What's your plan for this?

To assign this issue to you. ;)

We can use this for tracking.

[danzh2010]

I can work on this. But one question here, is there any reason why this even needs to be an option for TCP listener? Do we want different TCP listeners on different worker threads to listen on different sockets?

[mattklein123]

But one question here, is there any reason why this even needs to be an option for TCP listener? Do we want different TCP listeners on different worker threads to listen on different sockets?

It has been requested previously that we support this for TCP listeners also. In some cases this can lead to better performance in very high CPS use cases with many listening cores. IMO you don't need to implement it for TCP if you don't want to, but it would be nice to do it in such a way that someone could come along and do it for TCP later if they want?

[l8huang]

@danzh2010 Considering this issue assigned to you, what's your plane? :) Are you working on both UDP and TCP change? Or only UDP?

I recently did some LnP tests for HTTP traffic, one finding is: usually Envoy consumes <24 CPUs and occasionally Envoy consumes 46 CPUs(the node has 32 cores with Hyper-thread, that's 64 CPUs). Envoy run with 64 worker threads.

Making this change for TCP is in my task list, just want to confirm if we have any overlapping ;) I'm willing to take the TCP part.

[danzh2010]

Currently the one and only socket is passed to each listener on workder threads through Network:ListenerConfig. My plan is to change ListenerConfig to carry a socket factory instead of a socket. And create the socket during listener creation via the factory. UDP and TCP factory can have different socket management behavior.

As the first step, I will just refactor the code to pass around socket factory. TCP socket factory always provide the same socket, but UDP socket factory creates a new socket for each listener.

[l8huang]

@danzh2010 I did a quick change to make SO_REUSEPORT works for TCP listeners, and run a load test, connections distributed to worker threads every well(checked downstream_cx_total and downstream_cx_active in /stats output).

A draft PR #8884 created, if you have time to do a quick view, that could be every helpful, just want to confirm this change is on correct direction.

[danzh2010]

I just found that SO_REUSEPORT doesn't behave as expected in Mac OS. Mac doesn't dispatch UDP packets according to network 5 tuple within the SO_REUSEPORT socket group. For connection-oriented UDP traffic, this means we don't have stable connection as packets from the same connection can be sent to different sockets.