Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: 64511+ Concurrent TCP sessions (address already in use) #6176

Closed
gopherbot opened this issue Aug 17, 2013 · 7 comments

Comments

Projects
None yet
4 participants
@gopherbot
Copy link

commented Aug 17, 2013

by paul@vanbrouwershaven.com:

I'm currently trying to break the 64511 concurrent tcp sessions barrier in my GO
application but I encounter some unexpected behavior.

I have optimized my kernel for high tcp concurrency as suggested by Richard Jones on his
blog back in 2008 (see metabrew.com blog below). Unfortunately it doesn't have any
effect on the number of concurrent sessions.

As you have only 64511 unprivileged ports available per IP address, I have added several
additional IP addresses to my system and made go loop though these addresses. When I use
a LocalAddr with port 0 go directly starts complaining with "address already in
use" as soon I pass the +/- 64511 connections. So I tried to loop over my ip range
in combination with a specified port for the LocalAddr. Unfortunately this doesn't
improve the number of concurrent connections.

1 Million TCP connections (Linux):
http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3

2 Million TCP connections (FreeBSD):
http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/

I'm running a Ubuntu Server with a 3.2 kernel and go1.1.2 linux/amd64

My connection handling code:

RETRY:

// get the firs source ip:port address in queue 
ipNext = <-iprange

// manage our on port numbers per ip to overcome the 64511 limit
host, port, _ := net.SplitHostPort(ipNext)
nextPort, _ := strconv.Atoi(port)
nextPort++
if nextPort > 65535 {
    nextPort = 1024
}
    
// add this address with the next port number back to the end of the queue
iprange <- host +":"+ strconv.Itoa(nextPort)
            
d := net.Dialer{Timeout: 1*time.Second, Deadline: time.Now().Add(2*time.Second)}
d.LocalAddr, err = net.ResolveTCPAddr("tcp4", ipNext)
if err != nil {
        log.Fatal(err)
}
dc, err := d.Dial("tcp", dst)

if err != nil && strings.Contains(err.Error(), "address already in
use") {
    goto RETRY
}

--
See also the golang-nuts thread:
https://groups.google.com/forum/#!topic/golang-nuts/Mi7QkAqP7II
@remyoudompheng

This comment has been minimized.

Copy link
Contributor

commented Aug 17, 2013

Comment 1:

Please show the code of a complete program.
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Aug 17, 2013

Comment 2:

Outgoing connections also consume a port. Please show a complete working example.

Labels changed: removed go1.2maybe.

Status changed to WaitingForReply.

@gopherbot

This comment has been minimized.

Copy link
Author

commented Aug 19, 2013

Comment 3 by paul@vanbrouwershaven.com:

Here is a demo code that is running 60k concurrent connections with a timeout of 2
seconds.
http://play.golang.org/p/mQZ2nlUQsr
Please note that you need to increase your file descriptors as below and you a multicore
system to properly run this code!
sysctl -w fs.file-max=999999
ulimit -n `cat /proc/sys/fs/file-max`
The code will print a status output like below every 5 seconds:
2013/08/19 10:51:33 GO: 60007 || TCP: ESTABLISHED 4 SYN_SENT 28232   || ERROR: dial tcp
10.2.149.179:80: address already in use
/tmp/count.sh contains a count on current open tcp sessions:
#!/bin/sh
netstat -n | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state, t[state]} }'
@gopherbot

This comment has been minimized.

Copy link
Author

commented Aug 19, 2013

Comment 4 by paul@vanbrouwershaven.com:

Here is a demo code that is running 60k concurrent connections with a timeout of 2
seconds.
http://play.golang.org/p/mQZ2nlUQsr
Please note that you need to increase your file descriptors as below and a multicore
system to properly run this code!
sysctl -w fs.file-max=999999
ulimit -n `cat /proc/sys/fs/file-max`
The code will print a status output like below every 5 seconds:
2013/08/19 10:51:33 GO: 60007 || TCP: ESTABLISHED 4 SYN_SENT 28232   || ERROR: dial tcp
10.2.149.179:80: address already in use
/tmp/count.sh contains a count on current open tcp sessions:
#!/bin/sh
netstat -n | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state, t[state]} }'
@gopherbot

This comment has been minimized.

Copy link
Author

commented Sep 3, 2013

Comment 6 by paul@vanbrouwershaven.com:

This problem is kernel related. By specifying the ports in LocalAddr you can indeed
bypass the kernel problem. The actual problem in my code was 'solved' by removing a go
channel that was counting the number of cross routine completed connections,
unfortunately connections where completing faster than the channel could process them.
Increasing the channel buffer was only delaying the problem.
While you still have to take care of the "address already in use" error as it will
happen sometime when you select a port that is used by a tcp session outside the go
program.
With the code from the first post in this issue you can make millions of concurrent tcp
connections with golang.
@gopherbot

This comment has been minimized.

Copy link
Author

commented Nov 11, 2013

Comment 7:

Instead of picking source addresses in the app, you should be able to configure your
routing to pick from multiple source addresses. On linux, iproute2's "ip ro add ... src
..." and "equalize" options should prove useful, or go for policy routing if you must.
It'll still pick a source address first, and then try to find a free port, so the
balancing is stochastic at best.
But this seems better to me than putting all this logic in the app.
@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 11, 2013

Comment 8:

From comment #6 it seems like everything is working as well as it could. The kernel is
getting in the way, but there's not a lot Go can do about that.

Status changed to WorkingAsIntended.

@golang golang locked and limited conversation to collaborators Jun 24, 2016

This issue was closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.