net: ResolveIPAddr triggers glibc bug writing to wrong fd

[gopherbot]

by arnaud.lb:

It seems that ResolveIPAddr() can write to random unrelated file descriptors in some
unknown conditions, at high concurrency.

The code at [1] does a lot a DNS requests in parallel. At the same time, it also does
short connections to a local unix socket (Dial, a few Writes, and Close).

After a few seconds, the unix socket's listener is seeing random garbage, presumably DNS
requests (I can sometimes see part of domain names in the garbage).

It looks very much like ResolveIPAddr() is writing to random file descriptor; or that it
is corrupting some buffer.

To reproduce, run ./server and ./client. The server should start receiving binary
garbage after a few seconds (and logs that to stderr).

go version go1.1.2 linux/amd64

[1] https://gist.github.com/arnaud-lb/3af01cdfb6b1ee38c122

[dvyukov]

Comment 1:

The symptoms sound very much like data race on file descriptor, when read/write races
with close.

[gopherbot]

Comment 2 by arnaud.lb:

Can this happen even if the net.Conn is not shared between goroutines?
go func() {
    conn, _ = net.Dial("unix", "/tmp/test.sock")
    conn.Write([]byte("foo\n"))
    conn.Close()
}()
Is this something Go or I can fix? Is there a workaround?

[mikioh]

Comment 3:

I don't understand how could unix-domain stuff use DNS resolver. It that a special
behavior of Linux or Linux runtime libraries?

[robpike]

Comment 4:

I agree that dvyukov's analysis is likely correct. Does the race detector pick up the
problem?

Labels changed: added priority-soon, go1.2, removed priority-triage.

Status changed to Accepted.

[mikioh]

Comment 5:

Sorry, I misunderstood.
Can you please try with the CGO-disabled client?
I just tried it a few minutes and got no garbage at listener side.
Probably the root cause is the same as issue #6232.

[gopherbot]

Comment 6 by arnaud.lb:

It reproduces easily with a large /proc/sys/net/core/somaxconn (e.g. 10K).
With CGO-disabled, the problem disappears.
Tried with -race; i had to reduce the number of goroutines to 2K and 6K for dns and
socket, so as to not hit the limit of 8192 threads. It doesn't seem to detect a problem,
however the bug doesn't reproduce either.

[remyoudompheng]

Comment 7:

We encounter the same problem in production, but I couldn't write a minimal example. It
seems related to the behaviour of the GNU libc, although I couldn't understand what
could be the issue. Maybe a TLS corruption? The libc caches and reuses file descriptors
of UDP sockets in a thread-local structure.

[ianlancetaylor]

Comment 8:

If it's related to TLS corruption or some sort of glibc bug, it ought to be possible to
reproduce the problem with a multi-threaded C program.

[gopherbot]

Comment 9 by arnaud.lb:

The attached bug6336.c reproduces the bug too. Must be a getaddrinfo() bug.

Attachments:

1. bug6336.c (4045 bytes)

[remyoudompheng]

Comment 10:

Thanks, I can use the program to reproduce the issue too, even with as low as 20
threads. I use glibc 2.18 on Archlinux at home. But strangely I couldn't reproduce the
bug using res_query/nquery instead of getaddrinfo.

[mikioh]

Comment 11:

Platform:
- It happens on Linux, mostly
- It doesn't happen on FreeBSD
Go runtime:
- It happens when we use DNS resolver written in C via CGO, on both go1.1 and tip
C runtime:
- It happens when we use ??? libc on linux 
- It doesn't happen when we use libc: e.g., shlib version 7 libc on freebsd. It contains
traditional DNS resolver from Berekely and getaddrinfo from KAME
Hm, what a mess.

[gopherbot]

Comment 12 by sebastien.paolacci:

Arnaud's reproducer does trigger the bug on both eglibc 2.6.11 and 2.6.13 (Debian
Squeeze & Wheezy).

[gopherbot]

Comment 13 by arnaud.lb:

This bug is crazy. I've filled a debian bug:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=722075

[davecheney]

Comment 14:

Thank you for taking it to (one of) the upstream(s). Given the number of systems
affected, it sounds like Go will need to develop its own workaround regardless.

[rsc]

Comment 15:

Without knowing more about exactly what the bug is in glibc,
the only workaround I can think of is to stop using cgo for networking on Linux.
That's possible but probably too big a hammer.