Add SO_REUSEPORT support for EntryPoints

[aofei]

Welcome!

• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you expect to see?

Somehow people never discuss SO_REUSEPORT here, but IMHO it can be very useful sometimes for a program like Traefik running on the very edge.

For example, if your Traefik occupies ports 80 and 443, then you have no way to gracefully upgrade or restart (currently the only way to reload the config) your Traefik without any service downtime. But with SO_REUSEPORT support, you can do canary deployments against Traefik. Just like its name, we can have two Traefik processes listen to the same TCP/UDP port (or, REUSE the same port).

Nginx supports reuseport, and it served us very well:

reuseport
this parameter (1.9.1) instructs to create an individual listening socket for each worker process (using the SO_REUSEPORT socket option on Linux 3.9+ and DragonFly BSD, or SO_REUSEPORT_LB on FreeBSD 12+), allowing a kernel to distribute incoming connections between worker processes. This currently works only on Linux 3.9+, DragonFly BSD, and FreeBSD 12+ (1.15.1).

I think bringing SO_REUSEPORT into Traefik will be very easy, given that there are only a few net.Listen* calls here: https://github.com/search?q=repo%3Atraefik%2Ftraefik+language%3Ago+NOT+path%3A*_test.go+%2Fnet%5C.Listen%5Cw*%5C%28%2F&type=code.

If the Traefik Team thinks it's good but doesn't have the bandwidth, I can help draft a PR. 🙂

[mpl]

Hello @aofei ,

That's an interesting idea, but as often, we would probably wait to gauge the interest of the community before spending some time on it.
So be aware that even if you open a PR, we might not examine it before we're ready to do so.

However, I am personally interested in how you would do that. In a few words, how would you change the underlying net listeners in traefik so that they have SO_REUSEPORT enabled?

thanks.

[aofei]

Hi @mpl!

how would you change the underlying net listeners in traefik so that they have SO_REUSEPORT enabled?

Just to be clear we are not replacing any existing net.Listen* calls, they're here to stay. In other words SO_REUSEPORT should definitely not be a default enabled feature. Because even though it's kernel magic, I believe it still has a performance hit, even negligible. Also, I suggest we only add support for the Linux platform, since the Linux kernel has the best support for SO_REUSEPORT.

What's off the top of my head is that we could introduce a new configuration option for EntryPoints, maybe a boolean static.EntryPoint.ReusePort (defaults to false).

## Static configuration
entryPoints:
  web:
    address: ":80"
    reusePort: true

We can then decide whether to use our own net.ListenConfig according to the static.Traefik.ReusePort. I believe there are only 3 main places to touch:

1. traefik/pkg/server/server_entrypoint_tcp.go

   Line 433 in 8174860

   listener, err := net.Listen("tcp", entryPoint.GetAddress())

2. traefik/pkg/server/server_entrypoint_tcp_http3.go

   Line 36 in 8174860

   conn, err := net.ListenPacket("udp", configuration.GetAddress())

3. traefik/pkg/udp/conn.go

   Line 41 in 8174860

   conn, err := net.ListenUDP(network, laddr)

As for the actual implementation of the SO_REUSEPORT-enabled net.ListenConfig, it's actually quite simple (for Linux):

// (I know this seems too simple to be true. But that's all the code that needs to be written.)
//
// Usage:
//     tcp80Listener, err := reusePortListenConfig.Listen(context.Background(), "tcp", "127.0.0.1:80")
//     tcp443Listener, err := reusePortListenConfig.Listen(context.Background(), "tcp", "127.0.0.1:443")
//     udp443PacketConn, err := reusePortListenConfig.ListenPacket(context.Background(), "udp", "127.0.0.1:443")
//     udp53PacketConn, err := reusePortListenConfig.ListenPacket(context.Background(), "udp", "127.0.0.1:53")
//     udp53Conn := udp53PacketConn.(*net.UDPConn)
var reusePortListenConfig = net.ListenConfig{
	Control: func(_, _ string, c syscall.RawConn) error {
		var err error
		c.Control(func(fd uintptr) {
			err = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEADDR, 1)
			if err != nil {
				return
			}
			err = unix.SetsockoptInt(int(fd), unix.SOL_SOCKET, unix.SO_REUSEPORT, 1)
			if err != nil {
				return
			}
		})
		return err
	},
}

[mpl]

Ah, the last part (3.) is what I was missing. I didn't remember/know that ListenConfig existed, thanks.

So, in 1. , you would not do a net.Listen anymore, but rather a reusePortListenConfig.Listen instead, right?

[aofei]

So, in 1. , you would not do a net.Listen anymore, but rather a reusePortListenConfig.Listen instead, right?

If static.EntryPoint.ReusePort is true, then yes.

Actually the net.Listen is just a shorthand for (&net.ListenConfig{}).Listen (https://cs.opensource.google/go/go/+/refs/tags/go1.20.3:src/net/dial.go;l=707-710). So we can add a static.EntryPoint.ListenConfig() to make things easier (in 1, entryPoint.ListenConfig().Listen("tcp", entryPoint.GetAddress())). Anyway, adding the SO_REUSEPORT gizmo to Traefik won't be too invasive to the codebase.

[aofei]

Hi again @mpl! Just got some time, so I wrote an implementation and opened a PR for it. (#9834)

Just to be clear, I get what you said about gauging the interest of the community first, and I respect that. I wrote the implementation just to show the Traefik Team how simple it is, and most importantly, how non-invasive it is. Hope this helps in the future when you decide whether to add this support.

[jpds]

@aofei Hello, thanks for this - I was looking for this myself earlier this week. Could you possibly adjust your patch so that it uses SO_REUSEPORT_LB on FreeBSD - this is closer to what Linux does for the same option.

[aofei]

@jpds Done. Thanks for the reminder, somehow I missed it.

[philipcristiano]

I would love to see SO_REUSEPORT added into Traefik. When updating configuration (using Consul-Template/Nomad) a new job/deploy is used and currently results in stopping/starting Traefik. Adding SO_REUSEPORT would allow updates with a much lower risk of downtime and an overall simpler configuration and deployment in this situation.

[traefiker]

Closed by #9834.