New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/ld: invoke host linker for Windows cgo build #4069

Closed
rsc opened this Issue Sep 12, 2012 · 54 comments

Comments

Projects
None yet
@rsc
Contributor

rsc commented Sep 12, 2012

It should be possible to invoke the host linker for a cgo build.
That is, we'd run 6l to write a .o file for the Go pieces and then invoke gcc to do the
final link with the .syso files. This would fix many corner cases in 6l's support for
linking arbitrary ELF objects into the binary.

I would like to keep 6l's arbitrary ELF support so that one need not have a C compiler
installed to use a Go compiler, but we need the 'invoke gcc' plan as a fallback.

6l = 6l, 5l, 8l here, to avoid confusion with host ld.
@rsc

This comment has been minimized.

Contributor

rsc commented Sep 12, 2012

Comment 1:

Issue #3591 has been merged into this issue.

@rsc

This comment has been minimized.

Contributor

rsc commented Dec 10, 2012

Comment 8:

Labels changed: added size-l.

@rsc

This comment has been minimized.

Contributor

rsc commented Dec 10, 2012

Comment 9:

Labels changed: added size-xl.

@rsc

This comment has been minimized.

Contributor

rsc commented Jan 7, 2013

Comment 10:

Labels changed: removed size-l.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 11, 2013

Comment 12:

Issue #4272 has been merged into this issue.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 11, 2013

Comment 13:

Issue #1781 has been merged into this issue.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 11, 2013

Comment 14:

Issue #3141 has been merged into this issue.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 11, 2013

Comment 15:

Issue #3755 has been merged into this issue.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 11, 2013

Comment 16:

An update: the beginning of this is implemented for darwin, linux, and *bsd on amd64 and
386. You can play by invoking the go build of a command with -ldflags=-hostobj but if it
breaks you get to keep both pieces.
Dwarf information is not correct, so you might need -ldflags='-w -hostobj' depending on
how paranoid the system linker is.
Windows has not been attempted yet.
ARM has not been attempted yet.
There is a real possibility that ARM and Windows will be skipped for this go-round.
@minux

This comment has been minimized.

Member

minux commented Mar 11, 2013

Comment 17:

how much remains to be done for cmd/5l?
@rsc

This comment has been minimized.

Contributor

rsc commented Mar 11, 2013

Comment 18:

It's unclear to me how much is left in cmd/5l. The 6l and 8l linkers already had a
concept of a relocation and recorded the places that needed to be relocated, as part of
the standard linking, so that individual functions could be laid out separately and put
together only at the end. Adjusting them to emit the not-applied relocations was fairly
straightforward. 
5l does not have that flexibility: it lays out the entire code segment at once, because
I never changed it like I did 6l and 8l. So the required changes are more spread out
than in 6l/8l. Any part of the code layout in asm.c that uses ->pc or ->value is
suspect. However, the simplicity of the ARM instruction set might make it easier to find
these.
At the least, the two 'TODO: Use addrel' comments need to do that, although the one for
UNDEF could be changed into two instructions instead (MOVW $0, R0; BL R0). Also any case
listed in the table for C_SBRA or C_ADDR arguments, so cases 5, 11, 63, 64, 65, 68, 69,
74, 93, 94. It could be that most of the work is in addpool. It's all very unclear to me.
@rsc

This comment has been minimized.

Contributor

rsc commented Mar 12, 2013

Comment 19:

Issue #4782 has been merged into this issue.

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Mar 12, 2013

Comment 20:

What is the plan for the ELF TLS sections? I have a test here that verifies whether
__thread variables work in foreign code:
https://golang.org/cl/7578043/
It currently crashes, presumably because TLS sections are missing from the -hostobj
generated executable.
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Mar 12, 2013

Comment 21:

As far as I can see, the gc ld is going to have to create a SHT_TLS section and
appropriate TLS relocations.  I don't really see a way around that.
@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Mar 12, 2013

Comment 22:

Does that imply the relatively expensive call to __tls_get_addr in every stack size
prelude in many functions? I was hacking on support for generating .so files from Go,
but got stuck on this expensive need.
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Mar 12, 2013

Comment 23:

Calling __tls_get_addr would be required when compiling in PIC mode for code to go into
shared libraries.  However, that is not the normal mode and I don't see why we should
support it to start.  When not compiling PIC and not putting the code into a shared
library there is no need to call __tls_get_addr.  Instead, 6l would generate a
R_X86_64_TPOFF32 reloc on a "mov %fs:0,REG" instruction.  8l would generate a
R_386_TLE_LE reloc on a "mov %gs:0,REG" instruction.  Not sure about 5l; we may need a
__aeabi_read_tp call there for ARMv5.  For sufficiently new ARM processors we can use an
mrc instruction to get the value and a R_ARM_TLS_LE32 reloc to get the offset.
@rsc

This comment has been minimized.

Contributor

rsc commented Mar 12, 2013

Comment 24:

I control the link order, so I am hoping I can just pass the files to gcc
in the right order and keep my current slots without any relocations. :-)
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Mar 27, 2013

Comment 25:

ELF TLS is now handled correctly on i386 and amd64.
ARM and Windows are still not done.
@DanielMorsing

This comment has been minimized.

Contributor

DanielMorsing commented Apr 5, 2013

Comment 26:

Running the host linker means that the gdb python script automatically loaded.
Anyone know how to load it manually?
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Apr 5, 2013

Comment 27:

I'm not sure I entirely understand the question, but in gdb you can
type "source go/src/pkg/runtime/runtime-gdb.py".
@DanielMorsing

This comment has been minimized.

Contributor

DanielMorsing commented Apr 5, 2013

Comment 28:

Oh man, I messed up that sentence.
When host linking, the .debug_gdb_scripts symbol isn't emitted, so the gdb script isn't
automatically loaded.
Manually loading the script complains about a missing type symbol with the following
error:
Loading Go Runtime support.
Traceback (most recent call last):
  File "../go/src/pkg/runtime/runtime-gdb.py", line 198, in <module>
    _rctp_type = gdb.lookup_type("struct runtime.rtype").pointer()
gdb.error: No struct type named runtime.rtype.
@DanielMorsing

This comment has been minimized.

Contributor

DanielMorsing commented Apr 5, 2013

Comment 29:

Since the debug problem wont be solved before go1.1, I've created issue #5221.
@adg

This comment has been minimized.

Contributor

adg commented Apr 9, 2013

Comment 30:

The work related to Go 1.1 is done.

Labels changed: removed go1.1.

@rsc

This comment has been minimized.

Contributor

rsc commented Jul 30, 2013

Comment 31:

Labels changed: added go1.3.

@robpike

This comment has been minimized.

Contributor

robpike commented Aug 20, 2013

Comment 32:

Labels changed: removed go1.3.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Sep 5, 2013

Comment 33:

Issue #5106 has been merged into this issue.

@eliasnaur

This comment has been minimized.

Contributor

eliasnaur commented Sep 12, 2013

Comment 34:

issue #5590 added host linker support to 5l and added linux/arm to the list of os/arch
targets supporting host linking.
@rsc

This comment has been minimized.

Contributor

rsc commented Oct 29, 2013

Comment 35:

It sounds like we are just missing Windows support.
@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Oct 29, 2013

Comment 36:

> ... missing Windows support.
I never understood what is required - I don't use gcc myself. So, if someone can explain
what needs to be done (step by step), I will have a go at it.
Alex
@rsc

This comment has been minimized.

Contributor

rsc commented Oct 29, 2013

Comment 37:

I don't know exactly what is required. I plan to take a closer look once
some other linker cleanup is done for Go 1.3.

neclepsio added a commit to neclepsio/qml that referenced this issue Mar 4, 2015

Update capi.cpp
Removes all dynamic_cast, in order to make it work without RTTI. This is due to golang/go#10023 (duplicate of golang/go#4069) and solves go-qml#19.

neclepsio added a commit to neclepsio/qml that referenced this issue Mar 4, 2015

Update idletimer.cpp
Make it work without global variable initialization. This is due to golang/go#10023 (duplicate of golang/go#4069) and solves go-qml#142.

@minux minux assigned minux and unassigned rsc Mar 10, 2015

minux added a commit to minux/go-windows that referenced this issue Mar 10, 2015

cmd/internal/ld/pe.go: ask for 32-byte align on .data and .bss
ld was only providing 4-byte alignment, which is not enough even on
386 (runtime.test_z64 requires 8-byte alignment and was not met
before this change.)

Like 640 KB memory, 32-byte alignment should be enough for everybody.

misc/cgo/test passed with this change in external linking mode.

Update golang/go#4069: this change fixes the issue for 386.

Signed-off-by: Shenghou Ma <minux@golang.org>
@minux

This comment has been minimized.

Member

minux commented Mar 14, 2015

Just found another benefit of external linking.
the test misc/cgo/test, when internal linked, the size is 5112832 bytes, when external linked,
it's 1MB smaller, at 4025803 bytes. (This is on windows/amd64.)

@yinhm

This comment has been minimized.

yinhm commented Mar 16, 2015

Happy to see this will be fixed in Go1.5, really appreciating.
For the binary size, as long as it works on windows, we will be happy.

@chai2010

This comment has been minimized.

Contributor

chai2010 commented Mar 16, 2015

@minux Greate, thanks for your work!
As a windows gopher, we can use static link with the C++ libs!

minux added a commit that referenced this issue Mar 24, 2015

cmd/internal/ld, cmd/8l: external linking for windows/386
Update #4069: this CL fixes the issue on windows/386.

Signed-off-by: Shenghou Ma <minux@golang.org>
Change-Id: I2d2ea233f976aab3f356f9b508cdd246d5013e2e
Reviewed-on: https://go-review.googlesource.com/7283
Reviewed-by: Ian Lance Taylor <iant@golang.org>

@minux minux closed this in 4d59d1f Mar 24, 2015

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Apr 2, 2015

I compared file sizes for misc/cgo/test with -linkmode=internal and -linkmode=external. And the difference (about 1M) seems to be in dwarf sections (all sections starting with .debug). Should we do something about it?

Alex

@minux

This comment has been minimized.

Member

minux commented Apr 2, 2015

KellyLSB added a commit to HexxedUS/qml that referenced this issue May 28, 2016

Update capi.cpp
Removes all dynamic_cast, in order to make it work without RTTI. This is due to golang/go#10023 (duplicate of golang/go#4069) and solves go-qml#19.

KellyLSB added a commit to HexxedUS/qml that referenced this issue May 28, 2016

Update idletimer.cpp
Make it work without global variable initialization. This is due to golang/go#10023 (duplicate of golang/go#4069) and solves go-qml#142.

@golang golang locked and limited conversation to collaborators Jun 24, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.