net/http, x/net/netutil: Cannot Set TCP Connection Limitation for Go HTTP Server

[mcgradycchen]

What version of Go are you using (go version)?

1.13.4 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

What did you do?

I have a piece of code to implement a simple HTTP server with golang.

package main

import (
	"io"
	"log"
	"net/http"
	"net"
	"golang.org/x/net/netutil"
)

func main() {
	helloHandler := func(w http.ResponseWriter, req *http.Request) {
		io.WriteString(w, "Hello, world!\n")
	}

	mux := http.NewServeMux()

	handler := http.HandlerFunc(helloHandler)

	mux.Handle("/hello", handler)

	connectionCount := 20

	l, err := net.Listen("tcp", ":8000")

	if err != nil {
		log.Fatalf("Listen: %v", err)
	}

	defer l.Close()

	l = netutil.LimitListener(l, connectionCount)

	log.Fatal(http.Serve(l, mux))
}

Using wrk to test the connection limitation.
./wrk -t4 -c100 -d30s 'http://127.0.0.1:8000/hello'

What did you expect to see?

The number of established tcp connection should be limited to 20

What did you see instead?

netstat -an | grep 8000 | wc -l
The output is 102. It looks like the LimitListener does not make server reject the new incoming TCP connections when connection number is more than 20.

[tv42]

'netstat -an | grep 8000 | wc -l'
The output is 102. It looks like the LimitListener does not make server reject the new incoming TCP connections when connection number is more than 20.

In what state? Only open sockets are limited by anything inside the app.

[dmitshur]

netutil.LimitListener is documented as:

LimitListener returns a Listener that accepts at most n simultaneous connections from the provided Listener.

As far as I understand, your code makes it so that more than 20 connections can be accepted simultaneously. However, after some connections are accepted, more connections can be accepted, so http.Server may end up having more than 20 simultaneous connections.

Do you think there's a bug somewhere, or is this a feature request?

[mcgradycchen]

Hi @tv42

netstat -an |grep 8000 | wc -l
101

All status of these 100 sockets are ESTABLISHED.

[mcgrady@nj-mcgrady-ch4 ~]$ netstat -an |grep 8000 
tcp6      80      0 :::8000                 :::*                    LISTEN     
tcp6      49      0 10.64.164.118:8000      10.206.145.101:39028    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38992    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38986    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38963    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:39012    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:39021    ESTABLISHED
tcp6       0    131 10.64.164.118:8000      10.206.145.101:38955    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38942    ESTABLISHED
tcp6       0    131 10.64.164.118:8000      10.206.145.101:38938    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38991    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38988    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38967    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38999    ESTABLISHED
tcp6      49      0 10.64.164.118:8000      10.206.145.101:38945    ESTABLISHED
.........

[mcgradycchen]

netutil.LimitListener is documented as:

LimitListener returns a Listener that accepts at most n simultaneous connections from the provided Listener.

As far as I understand, your code makes it so that more than 20 connections can be accepted simultaneously. However, after some connections are accepted, more connections can be accepted, so http.Server may end up having more than 20 simultaneous connections.

Do you think there's a bug somewhere, or is this a feature request?

@dmitshur Thanks for your explanation. I think I need a feature that only 20 connections can be accepted simultaneously.

[rokkerruslan]

There are several reasons why you see a large connections number.

First, netstat shows structures for both inbound and outbound connections. For example, for two connections you will see:

tcp4      83      0  127.0.0.1.8000         127.0.0.1.62011        ESTABLISHED
tcp4       0      0  127.0.0.1.62011        127.0.0.1.8000         ESTABLISHED
tcp4       0      0  127.0.0.1.8000         127.0.0.1.62010        ESTABLISHED
tcp4       0      0  127.0.0.1.62010        127.0.0.1.8000         ESTABLISHED

One structure for the server, one for the client.

Secondly, netstat shows structures for connections in other states, for example TIME_WAIT:

tcp4       0      0  127.0.0.1.8000         127.0.0.1.62010        ESTABLISHED
tcp4       0      0  127.0.0.1.62010        127.0.0.1.8000         ESTABLISHED
tcp4       0      0  127.0.0.1.61933        127.0.0.1.8000         TIME_WAIT
tcp4       0      0  127.0.0.1.61934        127.0.0.1.8000         TIME_WAIT
tcp4       0      0  127.0.0.1.61935        127.0.0.1.8000         TIME_WAIT

In this case, we see only one living connection, but there are 5 rows.

Thirdly, there is a difference between the accept system call and the connection in the ESTABLISHED status. Roughly speaking, these are two different things. That is, when the SYN packet arrives on the listening socket, the system immediately responds to the SYM / ACK client and creates a structure for the new connection. But only when the application makes a system accept, the system will create a file descriptor associated with the connection so that the application will use it later for read / write.

This difference will be visible if you look at the number of open file descriptors for the application. Set the limit to 1 and send two requests simultaneously (yes, you need to set time.Sleep in the handler so that it does not end immediately):

We look at the number of connections, one for the listening socket, two for incoming connections:

tcp46      0      0  *.8000                 *.*                    LISTEN
tcp4       0      0  127.0.0.1.8000         127.0.0.1.62086        ESTABLISHED
tcp4       0      0  127.0.0.1.62086        127.0.0.1.8000         ESTABLISHED
tcp4       0      0  127.0.0.1.8000         127.0.0.1.62010        ESTABLISHED
tcp4       0      0  127.0.0.1.62010        127.0.0.1.8000         ESTABLISHED

But we see only one file descriptor associated with the connection on port 62086.

$ lsof -c ll
COMMAND  PID   USER   FD     TYPE             DEVICE   SIZE/OFF                NODE NAME
ll      9780 rokker    3u    IPv6 0x89184be7a6c9b195        0t0                 TCP *:irdmi (LISTEN)
ll      9780 rokker    5u    IPv6 0x89184be7a6c9ca15        0t0                 TCP localhost:irdmi->localhost:62086 (ESTABLISHED)

Only when the first connection finish, the application will make another accept call and process the second connection on port 62010:

$ lsof -c ll
COMMAND  PID   USER   FD     TYPE             DEVICE   SIZE/OFF                NODE NAME
ll      9780 rokker    3u    IPv6 0x89184be7a6c9b195        0t0                 TCP *:irdmi (LISTEN)
ll      9780 rokker    5u    IPv6 0x89184be7a6c9ca15        0t0                 TCP localhost:irdmi->localhost:62010 (ESTABLISHED)

With LimitListener you can adjust the number of file descriptors for the server, but you cannot adjust the number of ESTABLISHED connections for the system.

We can say that the system queues incoming connections and they are waiting for their accept call. I think your question is, can we somehow regulate this queue? Yes, look at the listen system call and the backlog parameter - https://linux.die.net/man/2/listen. backlog has its pitfalls and you should study this question more carefully.

To summarize, let’s say:

1. LimitListener works correctly, it does not create additional file descriptors and does not call accept more than once it is set.
2. The system establishes a connection, that is, it puts the connection into queue.
3. If you want to limit the number of connections in the queue, use the backlog.

P.S. https://stackoverflow.com/questions/10002868/what-value-of-backlog-should-i-use

[mcgradycchen]

@rokkerruslan thank you very very much. I got it.

I can set the backlog queue length by command sudo sysctl -w net.core.somaxconn=32

Anyway, thanks again for your help.