net: not possible to set TCP backlog parameter for listener

[nemirst]

What version of Go are you using (go version)?

go version go1.14.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/cppko/.cache/go-build"
GOENV="/home/cppko/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/cppko//Source/gocode"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/snap/go/current"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/snap/go/current/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build230275253=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I'm crossbuilding from Linux to Windows (but that is not important here). When 500 clients connect simultaneously to my Go program (TCP server) it fails to accept them all because backlog queue gets filled. I process them as fast as possible. As a workaround, on Linux I fixed this by setting net.core.somaxconn kernel parameter. But on Windows server (at least on Microsoft Windows Server 2019 Datacenter) I didn't find similar OS based parameter/solution. For Windows I had to rebuild Go from source after changing net.maxListenerBacklog to return custom backlog value (for me SOMAXCONN_HINT(500)=-500 worked): https://github.com/golang/go/blob/master/src/net/sock_windows.go#L16. There are other workarounds probably but this was done just to verify if setting this parameter fixes the problem.

What did you expect to see?

Some kind of way to set backlog parameter before listener starts to accept connections so that clients don't disconnect just because backlog is full in bursty scenarios.

What did you see instead?

Clients disconnected when backlog was full (size is 200 on my system).

[kostix]

The discussion which led to creation of this issue.

I'd try to rephrase the issue a bit so it becomes supposedly more understandable.

To figure out the size of the TCP listening backlog, the Go library code reads the relevant system setting once.
Linux (and several other supported platforms) do have such a setting, while others have not.
Windows currently falls into the latter category and so for it this setting is defined as syscall.SOMAXCONN, which is currently defined to be 0x7fffffff.
Still, as some guy behind Windows SDK suggests here, since Windows 8, there is actually a way to specify the backlog when executing a TCP listen syscall: this is done by passing a negative integer as the second argument; ostensibly, the sign tells the Winsock stack this is a direct backlog override. (It's still not clear whether Windows has a system-wide setting to control the cap on the backlog.)

What @nemirst suggests is to have a way to explicitly specify the backlog size parameter when creating a listening TCP socket — something akin to what has been implemented in #9661 for tweaking options of already created sockets.
This would solve the problem on Windows — a programmer could set a value they would like to use, and could be also useful on other platforms — to have more fine-grained control over listening sockets.

[ianlancetaylor]

In general we try to pass the maximum acceptable value as the listen backlog parameter. And it sounds like if we could do that on Windows, it would fix the problem that you are seeing.

From the docs (https://docs.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-listen), it looks like using SOMAXCONN sets a "maximum reasonable value." And we can instead use SOMAXCONN_HINT(N) where N is limited to be between 200 and 65535. So what if always use SOMAXCONN_HINT(65535)? It seems that that would fix your problem. Would it have any negative consequences?

[nemirst]

Yes, that would fix my problem completely without any real disadvantages for my case. I'm only not sure if that is appropriate for some other case, like:

1. In case allocated resources for socket are very important:

When calling the listen function in a Bluetooth application, it is strongly recommended that a much lower value be used for the backlog parameter (typically 2 to 4), since only a few client connections are accepted. This reduces the system resources that are allocated for use by the listening socket. This same recommendation applies to other network applications that expect only a few client connections.

2. If SYN flood attacks are possible
   https://tangentsoft.net/wskfaq/advanced.html, 4.14 - What is the connection backlog?

Beware that large backlogs make SYN flood attacks much more, shall we say, effective. When Winsock creates the backlog queue, it starts small and grows as required. Since the backlog queue is in non-pageable system memory, a SYN flood can cause the queue to eat a lot of this precious memory resource.
You will note that SYN attacks are dangerous for systems with both very short and very long backlog queues. The point is that a middle ground is the best course if you expect your server to withstand SYN attacks. Either use Microsoft’s dynamic backlog feature, or pick a value somewhere in the 20-200 range and tune it as required.

For me little bit higher memory usage is not important and also SYN flood attacks are not possible as server will not be public.

[acln0]

ISTM that it's not possible to use the ListenConfig.Control function to achieve this. Maybe we can add a backlog size knob to net.ListenConfig, then? If the knob is set (!= 0), we use that value. If it's not, we use the value we already use.

[ianlancetaylor]

@acln0 I agree that we could add a knob to ListenConfig, I just want to understand whether we really have to. My general impression of the backlog parameter to listen is that it has historical meaning but is pretty much useless on modern systems. Perhaps I am wrong.

[acln0]

@ianlancetaylor You may be right. I don't know about useless, but it's at least a little strange.

I implemented the knob I was talking about in package net, and then tried to write a test for it. The test tried to do something like this, on Linux:

lncfg := net.ListenConfig{Backlog: 1}
ln, _ := lncfg.Listen("tcp", ":0")

done1 := make(chan error)
go func() { done1 <- dial(ln.Addr()) }

time.Sleep(aShortWhile) // to give the first dialer time to race ahead, send a SYN, etc.

done2 := make(chan error)
go func() { done2 <- dial(ln.Addr()) }

select {
case err := <-done2:
	if err == nil {
		// bad
	}
case <-time.After(notVeryLong):
	// took too long, also bad
}

ln.Accept()
<-done1

Naively, I expected this to honor the backlog of 1. It didn't: both dials succeeded immediately. I don't know what this means, and I haven't had time to dig into the kernel sources in order to understand the behavior. Maybe Linux ignores backlog values that are this small.

On the other hand, leaving Linux aside for the moment, I am reading https://docs.microsoft.com/en-us/archive/blogs/winsdk/winsocks-listen-backlog-offers-more-flexibility-in-windows-8, which seems reasonably official, and mentions backlog parameters as low as 2 - 4, so maybe a test like that one would pass on Windows, and would also capture the essence of this issue (and make the knob worth doing).

[acln0]

If you think it would help, I could mail a CL of my implementation and the test (with DO NOT SUBMIT, etc.)

[gopherbot]

Change https://golang.org/cl/233577 mentions this issue: net: add Backlog knob to ListenConfig

[acln0]

I mailed it in the end. The relevant test is TestListenConfigBacklog, in listen_windows_test.go.

[matthewd98]

Change https://golang.org/cl/233577 mentions this issue: net: add Backlog knob to ListenConfig

This change never got merged unfortunately.

[ldemailly]

is this still the state of affairs? It seems ListenConfig Control should be able to change the raw socket... but you can't? (on unix)

[ianlancetaylor]

Adding a new parameter to ListenConfig should go through the proposal process. See https://go.dev/s/proposal.

[ldemailly]

I was a bit more curious about understanding why ListenConfig.Control can't be used at all to tweak the backlog (though a more direct api proposal sounds good too)

[ianlancetaylor]

ListenConfig.Control permits you to set socket options on the socket before calling syscall.Listen. But the backlog argument is passed to syscall.Listen itself, and ListenConfig.Control doesn't provide any mechanism for changing how syscall.Listen is called. As far as I know there is no socket option that can be used to set the backlog parameter and to ignore the backlog passed to syscall.Listen.

[ldemailly]

Thanks a lot for the reply and explanation, that gave me an idea for a workaround that at least on linux seems to work,
you can call syscall.Listen(fd, backlog) again later to update the backlog (obviously there is a race when you just start listening and it's not posix but ... it works)

[ldemailly]

In case it helps someone, pending the feature, wrote this (hack):

// backlog.Set() will attempt to reset the TCP connection backlog queue to the given value.
// Works on MacOS and Linux. On linux seems to allow n+1 connections to be queued (one in
// userland 1 in kernel maybe? despite no accept).
// To test
//   - run the server
//     go run sampleServer.go -b 3
//   - run the client with -n 5 only first 3 (or 4) will connect
//     go run sampleClient.go -n 5
//
// On linux `ss -ltn6` will show the backlog as SendQ column.
//
// PS: none of this meant to work according to POSIX, it just happens to seem to do so.\
// pending https://github.com/golang/go/issues/39000
func Set(l net.Listener, backlog int) error {
	tl, ok := l.(*net.TCPListener)
	if !ok {
		return fmt.Errorf("only tcp listener supported, called with %#v", l)
	}
	file, err := tl.File()
	if err != nil {
		return err
	}
	fd := int(file.Fd())
	err = syscall.Listen(fd, backlog)
	if err != nil {
		return err
	}
	return nil
}

You can find it here with the test programs: https://github.com/ldemailly/go-scratch/tree/main/backlog

[kostix]

func Set(l net.Listener, backlog int) error {
	tl, ok := l.(*net.TCPListener)
	if !ok {
		return fmt.Errorf("only tcp listener supported, called with %#v", l)
	}
	file, err := tl.File()
	if err != nil {
		return err
	}
	fd := int(file.Fd())
	err = syscall.Listen(fd, backlog)
	if err != nil {
		return err
	}
	return nil
}

net.TCPListener.File is documented to return a copy of the underlying FD:

func (l *TCPListener) File() (f *os.File, err error)
File returns a copy of the underlying os.File.
It is the caller's responsibility to close f when finished.
Closing l does not affect f, and closing f does not affect l.

<…>

(Emphasis mine.)

So, if I understand correctly, calling Set on a *TCPListener makes you end up with two listening sockets.

[ldemailly]

inode is the same so it does change the socket

[kostix]

inode is the same so it does change the socket

That is much better than I thought but still there's the problem that after a call to Set when you call Close on the *TCPListener being operated the socket won't really become closed. Or will it?

The same applies to messing with the socket parameters via the SyscallConn() method.

[ldemailly]

Not saying it’s a perfect solution, exposing backlog would be, yet meanwhile it can help someone. I would think listening sockets don’t really get closed until the server exits but I don’t know what happens if you need to close it before