Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: occasional failure with udp connections writing to each other #17677

Closed
vcabbage opened this issue Oct 30, 2016 · 3 comments

Comments

Projects
None yet
3 participants
@vcabbage
Copy link
Member

commented Oct 30, 2016

What version of Go are you using (go version)?

go version go1.7.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build464810642=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

What did you do?

Created two UDP connections. Sent a packet from conn A to conn B and from conn B to conn A, in separate goroutines.

Repo with details and steps to reproduce: https://github.com/vcabbage/go-net-repro

What did you expect to see?

No errors.

What did you see instead?

On most runs this would work. Occasionally one send operation fails with:
write udp4 0.0.0.0:60315->127.0.0.1:36289: sendto: operation not permitted

I have only seen the failure when running within Docker. I have tried running Docker with --privileged, --net=host, --set-cap=ALL and the error still occurs. I have also disabled the Docker host's firewall, as there are some references online to sendto returning EPERM when nf_conntack limit is hit.

excerpt from strace output:

[pid  5642] sendto(5, "", 0, 0, {sa_family=AF_INET, sin_port=htons(60315), sin_addr=inet_addr("127.0.0.1")}, 16 <unfinished ...>
[pid  5646] sendto(3, "", 0, 0, {sa_family=AF_INET, sin_port=htons(36289), sin_addr=inet_addr("127.0.0.1")}, 16 <unfinished ...>
[pid  5642] <... sendto resumed> )      = 0
[pid  5646] <... sendto resumed> )      = -1 EPERM (Operation not permitted)

(full output log in linked repo)

@quentinmit

This comment has been minimized.

Copy link
Contributor

commented Nov 1, 2016

I wonder if this is what happens when you have too many packets in the send buffer.

What does netstat -nu show for Recv-Q and Send-Q while the test is running?

@quentinmit quentinmit added this to the Go1.8Maybe milestone Nov 1, 2016

@vcabbage

This comment has been minimized.

Copy link
Member Author

commented Nov 1, 2016

@quentinmit netstat -nu doesn't display anything while the test is running. (Just the table headers.) I did notice that outgoing packets dropped in output of netstat -s increments every time I run the test.

Another point of information. I previously said I've only seen it fail in Docker. It also reliably fails on Ubuntu 14.04 (Parallels VM).

@vcabbage

This comment has been minimized.

Copy link
Member Author

commented Nov 2, 2016

This appears to be related to Docker and the kernel modules it loads. After removing Docker and rebooting, the host no longer exhibits the issue.

Not sure which one is the problem, doesn't seem to be hitting the nf_conntrack limit, but I think it's safe to say this isn't a Go bug.

For any curious, it seems to load all these:

aufs
bridge
ip_tables
ipt_MASQUERADE
iptable_filter
iptable_nat
llc
nf_conntrack
nf_conntrack_ipv4
nf_defrag_ipv4
nf_nat
nf_nat_ipv4
prl_tg
stp
x_tables
xfrm_algo
xfrm_user
xt_addrtype
xt_conntrack

@vcabbage vcabbage closed this Nov 2, 2016

@golang golang locked and limited conversation to collaborators Nov 2, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.