cmd/link: debugging fails with -linkmode external on macOS

[FiloSottile]

lldb HEAD works with linkmode internal on macOS Sierra:

$ go build -gcflags "-N -l" -x test.go
WORK=/var/folders/ry/v14gg02d0y9cb2w9809hf6ch0000gn/T/go-build450234285
mkdir -p $WORK/command-line-arguments/_obj/
mkdir -p $WORK/command-line-arguments/_obj/exe/
cd /Users/filippo/code/misc/rustgo
/usr/local/Cellar/go/1.8.1_1/libexec/pkg/tool/darwin_amd64/compile -o $WORK/command-line-arguments.a -trimpath $WORK -N -l -p main -complete -buildid 671e0e95e61f5bf7c157a090a3801fe9766c1a0a -D _/Users/filippo/code/misc/rustgo -I $WORK -pack ./test.go
cd .
/usr/local/Cellar/go/1.8.1_1/libexec/pkg/tool/darwin_amd64/link -o $WORK/command-line-arguments/_obj/exe/a.out -L $WORK -extld=clang -buildmode=exe -buildid=671e0e95e61f5bf7c157a090a3801fe9766c1a0a $WORK/command-line-arguments.a
mv $WORK/command-line-arguments/_obj/exe/a.out test

$ sudo /usr/local/opt/llvm/bin/lldb test
(lldb) target create "test"
Current executable set to 'test' (x86_64).
(lldb) b main.main
Breakpoint 1: where = test`main.main at test.go:5, address = 0x000000000104bf20
(lldb) run
Process 5724 launched: '/Users/filippo/code/misc/rustgo/test' (x86_64)
Process 5724 stopped
* thread #6, stop reason = breakpoint 1.1
    frame #0: 0x000000000104bf20 test`main.main at test.go:5
   2
   3   	package main
   4
-> 5   	func main() {
   6   		println("foo")
   7   	}

But fails with linkmode external:

$ go build -gcflags "-N -l" -ldflags "-linkmode external" -x test.go
WORK=/var/folders/ry/v14gg02d0y9cb2w9809hf6ch0000gn/T/go-build843720462
mkdir -p $WORK/command-line-arguments/_obj/
mkdir -p $WORK/command-line-arguments/_obj/exe/
cd /Users/filippo/code/misc/rustgo
/usr/local/Cellar/go/1.8.1_1/libexec/pkg/tool/darwin_amd64/compile -o $WORK/command-line-arguments.a -trimpath $WORK -N -l -p main -complete -buildid 09e299344f906f0b167907ec8bcdd857d3794aa0 -D _/Users/filippo/code/misc/rustgo -I $WORK -pack ./test.go
cd .
/usr/local/Cellar/go/1.8.1_1/libexec/pkg/tool/darwin_amd64/link -o $WORK/command-line-arguments/_obj/exe/a.out -L $WORK -extld=clang -buildmode=exe -buildid=09e299344f906f0b167907ec8bcdd857d3794aa0 -linkmode external $WORK/command-line-arguments.a
mv $WORK/command-line-arguments/_obj/exe/a.out test

$ sudo /usr/local/opt/llvm/bin/lldb test
(lldb) target create "test"
Current executable set to 'test' (x86_64).
(lldb) b main.main
Breakpoint 1: where = test`main.main, address = 0x000000000404d060
(lldb) run
Process 5854 launched: '/Users/filippo/code/misc/rustgo/test' (x86_64)
foo
Process 5854 exited with status = 0 (0x00000000)

This appears to be a regression to #8973

[FiloSottile]

Interestingly, it works with Delve in both cases.

[aarzilli]

If it really was a regression on #8973 lldb would complain differently. One possibility is that debug_info is mangled somehow, that would explain why delve appears to work (delve is probably reading the entry point from gosymtab, which lldb doesn't know about and an external linker wouldn't mangle). If I'm correct I wouldn't trust delve with this binary, it's probably going to fail in other ways.

IIRC things changed with regards to DWARF linking on macOS in go1.9, this could be already fixed.

[FiloSottile]

IIRC things changed with regards to DWARF linking on macOS in go1.9, this could be already fixed.

Sadly, reproduced with tip.

[aarzilli]

I get a different error on tip:

(lldb) target create "./test.extl"
Current executable set to './test.extl' (x86_64).
(lldb) b main.main
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
(lldb) 

however the binary has debug_info, debug_frame and debug_line and at a glance they all seem correct, I'm not sure what's lldb's problem here.

[FiloSottile]

For the record, I compiled lldb from master with brew install llvm --with-lldb --HEAD

[aarzilli]

Alright, lldb compiled from HEAD behaves like you say:

(lldb) target create "./test"
Current executable set to './test' (x86_64).
(lldb) b main.main
Breakpoint 1: where = test`main.main, address = 0x00000000040500d0
(lldb) r
Process 39532 launched: './test' (x86_64)
hello world
Process 39532 exited with status = 0 (0x00000000) 
(lldb) 

and setting log enable lldb state break and log enable dwarf all I get this:

warning: Tried to add breakpoint site at 0xffffffffffffffff but it was already present.

Warning: could not set breakpoint site for breakpoint location 1 of breakpoint 1.

So it's resolving the expression main.main to 0x00000000040500d0 which is (mostly) correct but then tries to set the breakpoint at 0xffffffffffffffff instead, for some reason. It looks like LLDB's bug.

[bradfitz]

Okay, closing this since it looks like an LLDB bug. Let us know if that's wrong.

[steeve]

For the record, I'm still getting this issue on LLDB shipped with Xcode 9:

$ lldb --version
lldb-900.0.45
  Swift-4.0
$ go version
go version go1.9.1 darwin/amd64
$ go build -x -gcflags="-N -l" -ldflags="-linkmode=external" github.com/sample/tst
$ xcrun lldb  ./tst
(lldb) target create "./tst"
Current executable set to './tst' (x86_64).
(lldb) log enable lldb state break
(lldb) log enable dwarf all
(lldb) b main.main
Target::AddBreakpoint (internal = no) => break_id = 1: name = 'main.main'


(x86_64) ./tst: SymbolFileDWARF::FindFunctions (name="main", name_type_mask=0x38, append=1, sc_list)
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
(lldb)

It works correctly with linkmode=internal.

[steeve]

I have some more information about this. In short, it seems the __debug_info section is wrong, which does not hint at a LLDB bug, so I believe this issue should be reopened.

Also, I tried as far back as go 1.5 and could not get it to work. I do believe this never worked.

I made a small test program, linked like this:

$ go build -x -o internal
$ go build -x -o external -ldflags "-linkmode external"

Some data:

$ objdump -section-headers internal
internal:   file format Mach-O 64-bit x86-64

Sections:
Idx Name          Size      Address          Type
  0 __text        0004e3bf 0000000001001000 DATA
  1 __rodata      00028440 000000000104f3c0 DATA
  2 __symbol_stub1 00000000 0000000001077800 TEXT
  3 __typelink    00000790 0000000001077800 DATA
  4 __itablink    00000008 0000000001077f90 DATA
  5 __gosymtab    00000000 0000000001077f98 DATA
  6 __gopclntab   0002e252 0000000001077fa0 DATA
  7 __nl_symbol_ptr 00000000 00000000010a7000 DATA
  8 __noptrdata   00000a18 00000000010a7000 DATA
  9 __data        000007a0 00000000010a7a20 DATA
 10 __bss         0001e1e8 00000000010a81c0 BSS
 11 __noptrbss    00002338 00000000010c63c0 BSS
 12 __debug_abbrev 00000116 00000000010c9000 DATA
 13 __debug_line  0000acc7 00000000010c9116 DATA
 14 __debug_frame 00009f9c 00000000010d3ddd DATA
 15 __debug_pubnames 00001dd1 00000000010ddd79 DATA
 16 __debug_pubtypes 00005fae 00000000010dfb4a DATA
 17 __debug_aranges 00000030 00000000010e5af8 DATA
 18 __debug_info  000228c3 00000000010e5b28 DATA

$ objdump -section-headers external
external:   file format Mach-O 64-bit x86-64

Sections:
Idx Name          Size      Address          Type
  0 __stubs       0000008a 0000000004001fde TEXT
  1 __stub_helper 000000f6 0000000004002068 TEXT
  2 __text        0004ebb8 0000000004002160 DATA
  3 __rodata      00028442 0000000004050d20 DATA
  4 __typelink    00000790 0000000004079180 DATA
  5 __itablink    00000008 0000000004079910 DATA
  6 __gosymtab    00000000 0000000004079918 DATA
  7 __gopclntab   0002e576 0000000004079920 DATA
  8 __cstring     000000ce 00000000040a7e96 DATA
  9 __const       00000008 00000000040a7f68 DATA
 10 __unwind_info 00000088 00000000040a7f70 DATA
 11 __got         00000010 00000000040a8000 DATA
 12 __nl_symbol_ptr 00000010 00000000040a8010 DATA
 13 __la_symbol_ptr 000000b8 00000000040a8020 DATA
 14 __noptrdata   00000a18 00000000040a80e0 DATA
 15 __data        00000830 00000000040a8b00 DATA
 16 __bss         0001e1e0 00000000040a9340 BSS
 17 __noptrbss    00002338 00000000040c7520 BSS
 18 __debug_line  0000ec95 00000000040d8000 DATA
 19 __debug_pubnames 00009277 00000000040e6c95 DATA
 20 __debug_pubtypes 00004dc1 00000000040eff0c DATA
 21 __debug_aranges 00003710 00000000040f4ccd DATA
 22 __debug_info  000156f4 00000000040f83dd DATA
 23 __debug_frame 0000a054 000000000410dad1 DATA
 24 __debug_abbrev 00000123 0000000004117b25 DATA
 25 __debug_str   0000e285 0000000004117c48 DATA
 26 __apple_names 00007b5c 0000000004125ecd DATA
 27 __apple_namespac 00000024 000000000412da29 DATA
 28 __apple_types 00004636 000000000412da4d DATA
 29 __apple_objc  00000024 0000000004132083 DATA

The __debug_lines sections looks the same in both:

$ objdump -macho -s -section=__debug_line internal | head -20
internal:
Contents of section __debug_line:
 10c9116 c3ac0000 02008412 00000101 fc0a0a00  ................
 10c9126 01010101 00000001 002f7573 722f6c6f  ........./usr/lo
 10c9136 63616c2f 43656c6c 61722f67 6f2f312e  cal/Cellar/go/1.
 10c9146 392e312f 6c696265 7865632f 7372632f  9.1/libexec/src/
 10c9156 72756e74 696d652f 616c672e 676f0000  runtime/alg.go..
 10c9166 00002f75 73722f6c 6f63616c 2f43656c  ../usr/local/Cel
 10c9176 6c61722f 676f2f31 2e392e31 2f6c6962  lar/go/1.9.1/lib
 10c9186 65786563 2f737263 2f72756e 74696d65  exec/src/runtime
 10c9196 2f737475 62732e67 6f000000 002f7573  /stubs.go..../us
 10c91a6 722f6c6f 63616c2f 43656c6c 61722f67  r/local/Cellar/g
 10c91b6 6f2f312e 392e312f 6c696265 7865632f  o/1.9.1/libexec/
 10c91c6 7372632f 72756e74 696d652f 74797065  src/runtime/type
 10c91d6 6b696e64 2e676f00 0000002f 7573722f  kind.go..../usr/
 10c91e6 6c6f6361 6c2f4365 6c6c6172 2f676f2f  local/Cellar/go/
 10c91f6 312e392e 312f6c69 62657865 632f7372  1.9.1/libexec/sr
 10c9206 632f7275 6e74696d 652f6174 6f6d6963  c/runtime/atomic
 10c9216 5f706f69 6e746572 2e676f00 0000002f  _pointer.go..../
 10c9226 7573722f 6c6f6361 6c2f4365 6c6c6172  usr/local/Cellar

 $ objdump -macho -s -section=__debug_line external | head -20
external:
Contents of section __debug_line:
 40d8000 91ec0000 02004a13 00000101 fc0a0a00  ......J.........
 40d8010 01010101 00000001 002f7573 722f6c6f  ........./usr/lo
 40d8020 63616c2f 43656c6c 61722f67 6f2f312e  cal/Cellar/go/1.
 40d8030 392e312f 6c696265 7865632f 7372632f  9.1/libexec/src/
 40d8040 72756e74 696d652f 616c672e 676f0000  runtime/alg.go..
 40d8050 00002f75 73722f6c 6f63616c 2f43656c  ../usr/local/Cel
 40d8060 6c61722f 676f2f31 2e392e31 2f6c6962  lar/go/1.9.1/lib
 40d8070 65786563 2f737263 2f72756e 74696d65  exec/src/runtime
 40d8080 2f737475 62732e67 6f000000 002f7573  /stubs.go..../us
 40d8090 722f6c6f 63616c2f 43656c6c 61722f67  r/local/Cellar/g
 40d80a0 6f2f312e 392e312f 6c696265 7865632f  o/1.9.1/libexec/
 40d80b0 7372632f 72756e74 696d652f 74797065  src/runtime/type
 40d80c0 6b696e64 2e676f00 0000002f 7573722f  kind.go..../usr/
 40d80d0 6c6f6361 6c2f4365 6c6c6172 2f676f2f  local/Cellar/go/
 40d80e0 312e392e 312f6c69 62657865 632f7372  1.9.1/libexec/sr
 40d80f0 632f7275 6e74696d 652f6174 6f6d6963  c/runtime/atomic
 40d8100 5f706f69 6e746572 2e676f00 0000002f  _pointer.go..../
 40d8110 7573722f 6c6f6361 6c2f4365 6c6c6172  usr/local/Cellar

The __debug_info does not, in fact, it looks bad on external:

$ objdump -macho -s -section=__debug_info internal | head -20
internal:
Contents of section __debug_info:
 10e5b28 bf280200 04000000 00000801 676f0016  .(..........go..
 10e5b38 00100001 00000000 c0f30401 00000000  ................
 10e5b48 00000000 2f557365 72732f73 74656576  ..../Users/steev
 10e5b58 652f676f 2f737263 2f676974 6875622e  e/go/src/github.
 10e5b68 636f6d2f 7a6e6c79 2f6c6c64 62746573  com/znly/lldbtes
 10e5b78 7400476f 20636d64 2f636f6d 70696c65  t.Go cmd/compile
 10e5b88 20676f31 2e392e31 00027275 6e74696d   go1.9.1..runtim
 10e5b98 652e6d65 6d686173 68300040 10000100  e.memhash0.@....
 10e5ba8 0000004b 10000100 00000001 9c010570  ...K...........p
 10e5bb8 00019c9e 41010005 6800049c 110822ae  ....A...h.....".
 10e5bc8 41010005 7e723200 049c1110 22ae4101  A...~r2.....".A.
 10e5bd8 00000272 756e7469 6d652e6d 656d6861  ...runtime.memha
 10e5be8 73683800 50100001 00000000 a9100001  sh8.P...........
 10e5bf8 00000000 019c0105 7000019c 9e410100  ........p....A..
 10e5c08 05680004 9c110822 ae410100 057e7232  .h.....".A...~r2
 10e5c18 00049c11 1022ae41 01000002 72756e74  .....".A....runt
 10e5c28 696d652e 6d656d68 61736831 3600b010  ime.memhash16...
 10e5c38 00010000 00000911 00010000 0000019c  ................

$ objdump -macho -s -section=__debug_info external | head -20
external:
Contents of section __debug_info:
 40f83dd f0560100 04000000 00000801 01000000  .V..............
 40f83ed 16a02100 04000000 00b50605 04000000  ..!.............
 40f83fd 00000000 00040000 00320000 00024900  .........2....I.
 40f840d 0000a021 00040000 0000ab21 00040000  ...!.......!....
 40f841d 0000019c 01035a00 0000019c afe80000  ......Z.........
 40f842d 035c0000 00049c11 0822b4e8 0000035e  .\.......".....^
 40f843d 00000004 9c111022 b4e80000 00026200  ......."......b.
 40f844d 0000b021 00040000 00000922 00040000  ...!......."....
 40f845d 0000019c 01035a00 0000019c afe80000  ......Z.........
 40f846d 035c0000 00049c11 0822b4e8 0000035e  .\.......".....^
 40f847d 00000004 9c111022 b4e80000 00027300  ......."......s.
 40f848d 00001022 00040000 00006922 00040000  ..."......i"....
 40f849d 0000019c 01035a00 0000019c afe80000  ......Z.........
 40f84ad 035c0000 00049c11 0822b4e8 0000035e  .\.......".....^
 40f84bd 00000004 9c111022 b4e80000 00028500  ......."........
 40f84cd 00007022 00040000 0000c922 00040000  ..p"......."....
 40f84dd 0000019c 01035a00 0000019c afe80000  ......Z.........
 40f84ed 035c0000 00049c11 0822b4e8 0000035e  .\.......".....^
... Truncated to 20 lines to but it goes on and on

[steeve]

gobjdump and dwarfdump correctly find the debug info. I don't know how though since objdump only shows me what looks like "garbage" data for the __debug_info section.

$ dwarfdump external | head -20
----------------------------------------------------------------------
 File: external (x86_64)
----------------------------------------------------------------------
.debug_info contents:

0x00000000: Compile Unit: length = 0x000156f0  version = 0x0004  abbr_offset = 0x00000000  addr_size = 0x08  (next CU at 0x000156f4)

0x0000000b: TAG_compile_unit [1] *
             AT_name( "go" )
             AT_language( DW_LANG_Go )
             AT_low_pc( 0x00000000040021a0 )
             AT_high_pc( 0x00000000040506b5 )
             AT_stmt_list( 0x00000000 )
             AT_comp_dir( "/Users/steeve/go/src/github.com/znly/lldbtest" )
             AT_producer( "Go cmd/compile go1.9.1" )

0x0000002d:     TAG_subprogram [2] *
                 AT_name( "runtime.memhash0" )
                 AT_low_pc( 0x00000000040021a0 )
                 AT_high_pc( 0x00000000040021ab )

$ gobjdump --dwarf=info external | head -20

external:     file format mach-o-x86-64

Contents of the .debug_info section:

  Compilation Unit @ offset 0x0:
   Length:        0x156f0 (32-bit)
   Version:       4
   Abbrev Offset: 0x0
   Pointer Size:  8
 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    <c>   DW_AT_name        : (indirect string, offset: 0x1): go
    <10>   DW_AT_language    : 22	(Go)
    <11>   DW_AT_low_pc      : 0x40021a0
    <19>   DW_AT_high_pc     : 0x40506b5
    <21>   DW_AT_stmt_list   : 0x0
    <25>   DW_AT_comp_dir    : (indirect string, offset: 0x4): /Users/steeve/go/src/github.com/znly/lldbtest
    <29>   DW_AT_producer    : (indirect string, offset: 0x32): Go cmd/compile go1.9.1
 <1><2d>: Abbrev Number: 2 (DW_TAG_subprogram)
    <2e>   DW_AT_name        : (indirect string, offset: 0x49): runtime.memhash0

[steeve]

I'm attaching both binaries in case people want to check.

package main

func sayCoucou(name string) string {
        return "coucou " + name
}

func main() {
        println(sayCoucou("seutive"))
}

binaries.tar.gz

[heschi]

Happened across this bug. It looks to me like it's still an issue, even with Go 1.10. Reopening.