New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: gdb command "goroutine 1 bt" fails on core file #17575

Open
kayuuzu opened this Issue Oct 25, 2016 · 12 comments

Comments

Projects
None yet
10 participants
@kayuuzu

kayuuzu commented Oct 25, 2016

What version of Go are you using (go version)?

go version go1.5.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/ldata/comp/project/go"
GORACE=""
GOROOT="/ldata/bin/go"
GOTOOLDIR="/ldata/bin/go/pkg/tool/linux_amd64"
GO15VENDOREXPERIMENT=""
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
CXX="g++"
CGO_ENABLED="1"

What did you do?

  1. export GOTRACEBACK=crash
  2. ulimit -c unlimited
  3. write a buggy program and run it to generate a core file
  4. use gdb to load core file
  5. source runtime-gdb.py
  6. run goroutine 1 bt

What did you expect to see?

print stack trace of goroutine 1

What did you see instead?

print error msg:
Python Exception <class 'gdb.error'> You can't do that without a process to debug.:
Error occurred in Python command: You can't do that without a process to debug.

@kayuuzu

This comment has been minimized.

kayuuzu commented Oct 25, 2016

runtime-gdb.py seems to set pc&sp registers when you run goroutine 1 bt, but registers are read-only after you load core file.

@quentinmit quentinmit changed the title from fail to print stack trace of goroutine when using core file to runtime: gdb command "goroutine 1 bt" fails on core file Oct 28, 2016

@quentinmit quentinmit added the NeedsFix label Oct 28, 2016

@quentinmit

This comment has been minimized.

Contributor

quentinmit commented Oct 28, 2016

I'm not sure what option we have here instead of setting $pc and $sp.

/cc @aclements @cherrymui

@quentinmit quentinmit added this to the Go1.9Maybe milestone Oct 28, 2016

@aclements

This comment has been minimized.

Member

aclements commented Oct 28, 2016

Unfortunately, before GDB 7.10, setting $pc and $sp is our only option. For GDB 7.10 and up, I believe we can use "unwinder" support, which is a much better way to do this. Not having GDB 7.10, I can't easily try this out. :) Ubuntu 16.04LTS has GDB 7.11, so this is probably becoming more widely available.

@aclements

This comment has been minimized.

Member

aclements commented Oct 28, 2016

BTW, if someone does want to try this out with GDB 7.10+, I think the code would be something like:

try:
    from gdb.unwinders import Unwinder
except ImportError:
    Unwinder = None

if Unwinder is not None:
    class FrameID(object):
        def __init__(self, sp, pc):
            self.sp = sp
            self.pc = pc

    class GoUnwinder(Unwinder):
        def __init__(self):
            super(GoUnwinder, self).__init___("go-unwinder")

        def __call__(pending_frame):
            # This only applies to the first frame.
            self.enabled = False

            # Ignore registers in pending_frame. Use stashed PC/SP.
            return pending_frame.create_unwind_info(self.frame_id)
    goUnwinder = GoUnwinder()

# ... in GoroutineCmd ...
goUnwinder.frame_id = FrameID(sp, pc)
goUnwinder.enabled = True
try:
    gdb.execute(cmd)
finally:
    goUnwinder.enabled = False

This is completely untested.

@aclements

This comment has been minimized.

Member

aclements commented Dec 21, 2016

FWIW, I recently wrote a gdb script that addresses this by copying the core file and rewriting the PC/SP saved in the core file for thread 1. You can then open this new core file and backtrace from it. It's awful, but it works (at least, on linux/amd64). https://gist.github.com/aclements/8d2d6a1d1ade4bc4fd492db6d3eb07a5

@kayuuzu

This comment has been minimized.

kayuuzu commented Dec 26, 2016

@aclements Good job. I will try it.

@mail2fish

This comment has been minimized.

mail2fish commented May 16, 2017

Is there any update for this issue?

@bradfitz

This comment has been minimized.

Member

bradfitz commented May 16, 2017

@mail2fish, any updates would be posted here. This is not happening for Go 1.9 apparently.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9Maybe May 16, 2017

@aclements

This comment has been minimized.

Member

aclements commented May 24, 2017

I got my hands on a more recent version of GDB that supports the frame unwinders API and completely failed to get it to do anything useful. If anyone wants to pick up where I left off, here's the code that monkey-patches in the unwinder: https://gist.github.com/aclements/e8b4b3d887ccc9fd2907e696a8b4b2a2. It gets into the custom unwinder, but GDB seems to ignore the updated unwind info it returns. My next step would probably be to run GDB under GDB and figure out what's going on inside.

@hyangah hyangah added the Debugging label Jul 18, 2017

@occia

This comment has been minimized.

occia commented Aug 24, 2017

So can I say, so far, there isn't a way to look the stack trace from a core dump of an exe written in Golang? I really need the answer, and a more detail question is at here thanks : )

@aclements

This comment has been minimized.

Member

aclements commented Aug 24, 2017

@occia, the closest thing we have right now is the hack in #17575 (comment). Unfortunately, this is a limitation of GDB and there's not much we can do about it. In theory a custom unwinder should make it possible, but all of my experiments with custom unwinding failed to produce any results at all.

@heschik

This comment has been minimized.

Contributor

heschik commented Oct 17, 2017

For posterity, dlv should work fine for this. dlv core <binary> <core file>.

@rsc rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017

@gopherbot gopherbot modified the milestones: Go1.11, Unplanned May 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment