Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: SetReadBuffer makes i/o very slow on OS X Mavericks #6930

Closed
gopherbot opened this issue Dec 10, 2013 · 15 comments

Comments

Projects
None yet
5 participants
@gopherbot
Copy link

commented Dec 10, 2013

by derek.collison:

I came across a sever performance regression on MacOSX Mavericks. This problem does not
exist on Mountain Lion. I reproduced with Go1.2 and Go1.1.2.

I have attached a benchmark test with one that works as expected and one that tries to
set the ReadBuffer size. The second test will never complete due to the net.Read() calls
stalling.

What is the expected output?

  test/benchmark completes

What do you see instead?

  Never completes.

Which compiler are you using (5g, 6g, 8g, gccgo)?

  6g

Which operating system are you using?

  MacOSX Mavericks

Which version are you using?  (run 'go version')

  go version go1.2 darwin/amd64

Please provide any additional information below.

Attachments:

  1. mavbug_test.go (1775 bytes)
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Dec 10, 2013

Comment 1:

What happens if you remove the DNS lookup in Dial ? Ie, return a TCPListener, then get
the TCPAddr from it and use DialTCP("tcp", nil, addr) ?
@gopherbot

This comment has been minimized.

Copy link
Author

commented Dec 11, 2013

Comment 2 by derek.collison:

We can try that, but I doubt it would make a difference in the bug..
@mikioh

This comment has been minimized.

Copy link
Contributor

commented Dec 11, 2013

Comment 3:

Seems like Mavericks has a couple of minor changes to the buffer
management/reconfiguration stuff of network subsystems. I'd like to suggest you to
change/adjust "var bufSize = 32768" to fit your needs properly and see what happens. I
just guess, on Mavericks we have fine-granularity control to the buffers inside the
kernel, it's veiled until Mountain Lion.

Labels changed: added os-macosx.

@gopherbot

This comment has been minimized.

Copy link
Author

commented Dec 11, 2013

Comment 4 by derek.collison:

That was the number I used in https://github.com/apcera/gnatsd, so was representative of
a real world performance bug.
@mikioh

This comment has been minimized.

Copy link
Contributor

commented Dec 12, 2013

Comment 5:

Just skimmed source tree of FreeBSD and found that: At some point of FreeBSD
10-CURRENT/9-STABLE mbuf autotuning stuff was added to the kernel; see
kern/uipc_socket.c and uipc_sockbuff.c. After the patches, we can observe the behavior
you pointed out on FreeBSD 9-STABLE and 10, OS X Mavericks (that's likely because
Mavericks uses a part of FreeBSD 10 as its BSD subsystem), like the following:
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0  65380 127.0.0.1.30213        127.0.0.1.58842        ESTABLISHED
tcp46      0      0 *.58842                *.*                    LISTEN
a completely squeezed Recv-Q by specifying SO_RCVBUF option. IIRC, FreeBSD 9.2 and
prior, OS X Mountain Lion and prior do extend Recv-Q even if we specify a very small
SO_RCVBUF. Also we know that the behavior of SO_RCVBUF/SNDBUF options is really platform
and protocol-dependent stuf. Seems there's nothing we can do for now (except adding
platform/protocol independent socket buffer management stuff into the
net/runtime/syscall packages). WDYT?

Labels changed: added os-freebsd.

@mikioh

This comment has been minimized.

Copy link
Contributor

commented Dec 12, 2013

Comment 6:

/supersede/
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp4       0      0 127.0.0.1.17424        127.0.0.1.42072        ESTABLISHED // receiver
tcp4       0  65380 127.0.0.1.42072        127.0.0.1.17424        ESTABLISHED // sender
tcp46      0      0 *.17424                *.*                    LISTEN
@mikioh

This comment has been minimized.

Copy link
Contributor

commented Dec 13, 2013

Comment 7:

And there's a significant difference btw Darwin (OS X Mavericks) and latest FreeBSD
kernels. The attached code uses a wildcard address on both passive and active TCP
endpoint holders. Please replace them with some specific address such as 127.0.0.1 and
see what happens. Probably it would be a workaround.
A dumb hypothesis: Mavericks implements some fancy networking feature that tries to find
out the best IP transport layer path on the fly working together with IP routing.
Conventionally a packet to the wildcard address 0.0.0.0 is routed to the localhost, and
it might be some cost on Mavericks.

Status changed to WaitingForReply.

@gopherbot

This comment has been minimized.

Copy link
Author

commented Dec 16, 2013

Comment 8 by derek.collison:

For now I have removed setting the Read or Write buffer size, and performance seemed to
be best even when I set the values to many different settings.
@mikioh

This comment has been minimized.

Copy link
Contributor

commented Dec 16, 2013

Comment 9:

Thanks for your cooperation. Yup, that would be a most stable workaround for now. I'll
try to run dtrace at every TCP state-change and IO probe points on Mavericks later,
probably new year holidays. I mean, I didn't see "test never completed" on latest
FreeBSD kernels, sigh.

Labels changed: removed os-freebsd.

Status changed to LongTerm.

@gopherbot

This comment has been minimized.

Copy link
Author

commented Dec 16, 2013

Comment 11 by derek.collison:

ok thanks..

@rsc rsc removed the os-macosx label Apr 10, 2015

@rsc

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2015

Really sounds like a Mavericks bug.

@rsc rsc changed the title darwin: the relationship btw kernel socket buffers and SO_SNDBUF/RCVBUF options net: SetReadBuffer makes i/o very slow on OS X Mavericks Apr 10, 2015

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc added the OS-Darwin label Apr 15, 2015

@danp

This comment has been minimized.

Copy link
Contributor

commented Nov 18, 2015

FWIW here is output from the attached benchmark using go1.5.1 on OS X 10.10.5 (Yosemite):

% go test -bench .
testing: warning: no tests to run
PASS
Benchmark_MavericksSocketIONoBuffer-4      50000         25333 ns/op    1293.47 MB/s
Benchmark_MavericksSocketIORegression-4     2000      15603790 ns/op       2.10 MB/s
ok      _/Users/dan.peterson/mavbug 32.904s
@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 18, 2015

I guess the interesting question is whether you can reproduce this in C or Python or something like that using plain setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &n, sizeof(int)) or whatever the incantation is.

Given that that's literally all SetReadBuffer does, it seems like it must be the kernel mishandling that. Unless our constants are wrong and what we think is SO_RCVBUF is really something like "set maximum bytes/second to read".

Still seems like a kernel bug. I guess we could make SetReadBuffer a no-op on OS X if we convinced ourselves of that.

@danp

This comment has been minimized.

Copy link
Contributor

commented Nov 18, 2015

Perhaps this is working as expected now? The benchmark is completing whereas it was not in the original report.

The size in the attached benchmark sets the buffer to 32k. Changing it to something larger (like 1M) yields results more in line with the default case:

Benchmark_MavericksSocketIONoBuffer-4       3000        392709 ns/op    2670.11 MB/s
Benchmark_MavericksSocketIORegression-4     5000        378669 ns/op    2769.10 MB/s
@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 18, 2015

OK, great, that does look fixed.

@rsc rsc closed this Nov 18, 2015

@golang golang locked and limited conversation to collaborators Nov 17, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.