Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: dns lookup crashes app : __libc_res_nquery: Assertion `hp != hp2' failed. #7191

Closed
gopherbot opened this issue Jan 23, 2014 · 9 comments

Comments

Projects
None yet
7 participants
@gopherbot
Copy link

commented Jan 23, 2014

by matthew.kanwisher:

Crash during dns lookups

nitro: res_query.c:251: __libc_res_nquery: Assertion `hp != hp2' failed.
SIGABRT: abort
PC=0x7fdc40d1c945
signal arrived during cgo execution

runtime.cgocall(0x4051f0, 0x7fdc197a0c70)
  /usr/local/go/src/pkg/runtime/cgocall.c:149 +0x11b fp=0x7fdc197a0c48
net._C2func_getaddrinfo(0x7fdc10909010, 0x0, 0xc2dbfda570, 0xc28f37e018, 0x7fdc197a0d08,
...)
  net/_obj/_cgo_defun.c:50 +0x36 fp=0x7fdc197a0c70
net.cgoLookupIPCNAME(0xc297cd4a80, 0x18, 0x0, 0x0, 0x0, ...)
  /usr/local/go/src/pkg/net/cgo_unix.go:96 +0x174 fp=0x7fdc197a0d48
net.cgoLookupIP(0xc297cd4a80, 0x18, 0x0, 0x0, 0x0, ...)
  /usr/local/go/src/pkg/net/cgo_unix.go:148 +0x69 fp=0x7fdc197a0da0
net.lookupIP(0xc297cd4a80, 0x18, 0x0, 0x0, 0x0, ...)
  /usr/local/go/src/pkg/net/lookup_unix.go:64 +0x63 fp=0x7fdc197a0de8
net.func·022(0x1740410, 0xc21060a690, 0xc297cd4a80, 0x18)
  /usr/local/go/src/pkg/net/lookup.go:41 +0x2d fp=0x7fdc197a0e38
net.(*singleflight).Do(0x1740410, 0xc297cd4a80, 0x18, 0x7fdc197a0ed0, 0x0, ...)
  /usr/local/go/src/pkg/net/singleflight.go:45 +0x1de fp=0x7fdc197a0e80
net.lookupIPMerge(0xc297cd4a80, 0x18, 0x0, 0x0, 0x0, ...)
  /usr/local/go/src/pkg/net/lookup.go:42 +0xc0 fp=0x7fdc197a0f30
net.func·023()
  /usr/local/go/src/pkg/net/lookup.go:80 +0x36 fp=0x7fdc197a0fa0
runtime.goexit()
  /usr/local/go/src/pkg/runtime/proc.c:1394 fp=0x7fdc197a0fa8
created by net.lookupIPDeadline
  /usr/local/go/src/pkg/net/lookup.go:82 +0x239

What steps will reproduce the problem?
If possible, include a link to a program on play.golang.org.
Not clear what steps reproduce

What is the expected output?
program shouldn't crash maybe just log an error


What do you see instead?
program crashes on uncaught error

Which compiler are you using (5g, 6g, 8g, gccgo)?
GO 1.2 with c libraries

Which operating system are you using?
linux

Which version are you using?  (run 'go version')
go version go1.2 linux/amd64

Please provide any additional information below.
Go lang should really catch exceptions on go routines in the underlying libraries
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2014

Comment 1:

Which operating system and libc version are you using ?

Labels changed: added release-none, repo-main.

Status changed to WaitingForReply.

@gopherbot

This comment has been minimized.

Copy link
Author

commented Jan 23, 2014

Comment 2 by matthew.kanwisher:

SUSE Linux Enterprise Server 11 (x86_64)
VERSION = 11
PATCHLEVEL = 2
(gnu libc) 2.11.3
@gopherbot

This comment has been minimized.

Copy link
Author

commented Jan 23, 2014

Comment 3 by matthew.kanwisher:

I'll mention a few things, we are running on quite a bit of load for 3 weeks now without
seeing a crash, we recently upgraded to golang 1.2 4 days ago, not sure if its a
regression or we just unluckily hit this exception.
@gopherbot

This comment has been minimized.

Copy link
Author

commented Jan 23, 2014

Comment 4 by matthew.kanwisher:

It appears to be a glibc issues ;( https://bugzilla.redhat.com/show_bug.cgi?id=730856 
you think if we put a recover in the goroutine it would prevent the app from crashing or
is there nothing go can do ?
@davecheney

This comment has been minimized.

Copy link
Contributor

commented Jan 23, 2014

Comment 5:

Good question. I am not sure if you can catch that type of panic, and I would be
concerned what state that would leave the cgo parts in. 
SLES 11 is current and supported, you should step on your support contacts there, you're
paying for it.

Status changed to Accepted.

@minux

This comment has been minimized.

Copy link
Member

commented Jan 23, 2014

Comment 6:

in general, we can't do much about signals received when executing cgo code.
1. we can't reliably unwind cgo code.
2. C is not memory safe, so by the time the signal arrives, the memory could have
already been
corrupted, for example, in this case, the fact that the glibc developers placed an
assertion in their
code probably suggests that if the assertion turns out to be false, it's internal is
inconsistent, so
even if we could recover from it, the next DNS lookup might trigger the problem again.
In other words, we can't do much about this issue (even C/C++ application can't do much
on this
issue)
If you can't deploy updates to the glibc bug, and can withstand the limits of the pure
Go resolver,
you can do this to recompile a pure Go version of net package without disabling cgo:
go install -tags netgo -a std
and then rebuild your program, so it won't use the buggy glibc resolvers (please be
aware that
the pure Go resolver has several limitations, and others probably could explain them
better than
me).
@gopherbot

This comment has been minimized.

Copy link
Author

commented Jan 23, 2014

Comment 7 by matthew.kanwisher:

Yeah we can't easily backport it, we are in the process of upgraded OSes. If anyone else
knows the limitations of the dns resolver that would be amazing to know.

@mikioh mikioh added help wanted and removed accepted labels Feb 4, 2015

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@rsc rsc removed release-none labels Apr 10, 2015

@danp

This comment has been minimized.

Copy link
Contributor

commented Nov 17, 2015

Given this issue's age and the fact it was a glibc issue that should have been fixed in SUSE and other distributions, should it be closed?

@bradfitz

This comment has been minimized.

Copy link
Member

commented Nov 17, 2015

Closing works for me. Plus Go uses its own DNS resolver by default these days.

@bradfitz bradfitz closed this Nov 17, 2015

@golang golang locked and limited conversation to collaborators Nov 16, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.