REUSE_PORT: define in Linux

[miekg]

Set SO_REUSEADDR on the UDP socket for improved performance.

[miekg]

This came up in coredns/perf-tests#4
cc @andrewtj @asergeyev could you take a look? Thanks!

[miekg]

Hmmm, in isolation the test works:

% go test -v -run TestShutdownUDP                                                                                             ~/g/src/github.com/miekg/dns so_reuseport
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS

[miekg]

something, something, reuse:

% for i in {0..10}; do go test -v -run TestShutdownUDP; done                                                                  ~/g/src/github.com/miekg/dns so_reuseport
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS
ok  	github.com/miekg/dns	1.012s
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS
ok  	github.com/miekg/dns	1.010s
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS
ok  	github.com/miekg/dns	1.015s
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS
ok  	github.com/miekg/dns	1.011s
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS
ok  	github.com/miekg/dns	1.009s
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS
ok  	github.com/miekg/dns	1.010s
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS
ok  	github.com/miekg/dns	1.010s
=== RUN   TestShutdownUDP
--- PASS: TestShutdownUDP (0.00s)
PASS
ok  	github.com/miekg/dns	1.023s
=== RUN   TestShutdownUDP
--- FAIL: TestShutdownUDP (2.00s)
	server_test.go:573: Could not shutdown test UDP server. Gave up waiting
FAIL
exit status 1
FAIL	github.com/miekg/dns	2.011s
=== RUN   TestShutdownUDP
--- FAIL: TestShutdownUDP (2.00s)
	server_test.go:573: Could not shutdown test UDP server. Gave up waiting
FAIL
exit status 1
FAIL	github.com/miekg/dns	2.017s
=== RUN   TestShutdownUDP
--- FAIL: TestShutdownUDP (2.00s)
	server_test.go:573: Could not shutdown test UDP server. Gave up waiting
FAIL
exit status 1
FAIL	github.com/miekg/dns	2.024s

[miekg]

For some weird reason the srv.inFlight sync.WaitGroup is not zero once we reach the Shutdown() code. I have no idea why.

[asergeyev]

Commenting setting reuse fixes it so somehow they really related....

[asergeyev]

http://lxr.free-electrons.com/source/include/uapi/asm-generic/socket.h#L25 where did you get constant from?

It's not failing with 15 for me but fails with 0x0200

[miekg]

It's not the WaitGroup (doing a inFlight.Done() in Shutdown() crashes with negative value every time).

[miekg]

@asergeyev ghe. I got the value from some page I can't find anymore :( Original patch going in kernel I think. Anyway seems to match values in Golang:

 grep -i REUSEPORT *                                                                                                                         ~/upstream/go/src/syscall
net_nacl.go:	SO_REUSEPORT
zerrors_darwin_386.go:	SO_REUSEPORT                      = 0x200
zerrors_darwin_amd64.go:	SO_REUSEPORT                      = 0x200
zerrors_darwin_arm64.go:	SO_REUSEPORT                      = 0x200
zerrors_darwin_arm.go:	SO_REUSEPORT                = 0x200
zerrors_dragonfly_amd64.go:	SO_REUSEPORT                      = 0x200
zerrors_freebsd_386.go:	SO_REUSEPORT                      = 0x200
zerrors_freebsd_amd64.go:	SO_REUSEPORT                      = 0x200
zerrors_freebsd_arm.go:	SO_REUSEPORT                      = 0x200
zerrors_linux_arm64.go:	SO_REUSEPORT                     = 0xf
zerrors_linux_mips64.go:	SO_REUSEPORT                     = 0x200
zerrors_linux_mips64le.go:	SO_REUSEPORT                     = 0x200
zerrors_linux_mips.go:	SO_REUSEPORT                     = 0x200
zerrors_linux_mipsle.go:	SO_REUSEPORT                     = 0x200
zerrors_linux_ppc64.go:	SO_REUSEPORT                     = 0xf
zerrors_linux_ppc64le.go:	SO_REUSEPORT                     = 0xf
zerrors_linux_s390x.go:	SO_REUSEPORT                     = 0xf
zerrors_netbsd_386.go:	SO_REUSEPORT                      = 0x200
zerrors_netbsd_amd64.go:	SO_REUSEPORT                      = 0x200
zerrors_netbsd_arm.go:	SO_REUSEPORT                      = 0x200
zerrors_openbsd_386.go:	SO_REUSEPORT                      = 0x200
zerrors_openbsd_amd64.go:	SO_REUSEPORT                      = 0x200
zerrors_openbsd_arm.go:	SO_REUSEPORT                      = 0x200

[asergeyev]

#include <sys/socket.h>
#include <stdio.h>

int main() {
    printf("%d\n", SO_REUSEPORT);
}

Prints 15 for me.

See http://lxr.free-electrons.com/ident?i=SO_REUSEPORT risc stuff says 0x200, others set to 15

I don't know what OSX would do.

[miekg]

Hmmm. Ghe :) Good be. Then interesting why:
https://gist.github.com/miekg/e80d2eadfea48422178cdbaaf9f8a33f
would speed CoreDNS up. Only the REUSEADDR?

I'll double check. Also try your value. Seems legit.

[miekg]

Also, also what does 0x200 do then???

[asergeyev]

It's not my value, it's what 4.10.9-200.fc25.x86_64 printed for me. I don't know kernel deep enough to know what would happen upon receiving 0x200. I'd expect some error and in case of test C program I actually get ENOPROTOOPT

[asergeyev]

Really not adding anything but it's worth to review:
http://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t

[miekg]

Confused about the figures... Now everything on my machine is fast and I can't reproduce the original case of getting 45K qps :/

Running go test -run='___' -bench BenchmarkServe -v does not indicate any speedup between this and master

[asergeyev]

BenchmarkServe should not be changed since it's single goroutine that reads from single socket handle.

[miekg]

BenchmarkServe should not be changed since it's single goroutine that reads from single socket handle.
Yeah, true. Only matters when we optimize internal code paths.

Thanks for double checking me on this whole thread @asergeyev

[miekg]

This doesn't help. Active perf testing of CoreDNS (and thus Go DNS) is happening in github.com/coredns/perf-tests