Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: support -linkmode=internal with Cgo #38918

Open
eliasnaur opened this issue May 7, 2020 · 4 comments
Open

cmd/link: support -linkmode=internal with Cgo #38918

eliasnaur opened this issue May 7, 2020 · 4 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@eliasnaur
Copy link
Contributor

eliasnaur commented May 7, 2020

What version of Go are you using (go version)?

$ go version
go version devel +3afa74115b Sun Apr 12 15:37:29 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/elias/.cache/go-build"
GOENV="/home/elias/.config/go/env"
GOEXE=""
GOFLAGS="-mod=readonly"
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/elias/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/elias/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/elias/dev/go-tip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/elias/dev/go-tip/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/elias/proj/gio/example/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build621404981=/tmp/go-build -gno-record-gcc-switches"

What did you do?

linux/amd64:

$ mkdir tmp
$ cd tmp
$ go mod init blah
go: creating new go.mod: module blah
$ go run -ldflags=-linkmode=internal gioui.org/example/kitchen
go: finding module for package gioui.org/example/kitchen
go: downloading gioui.org/example v0.0.0-20200506145455-3957be37c821
go: downloading gioui.org v0.0.0-20200506145455-3957be37c821
go: found gioui.org/example/kitchen in gioui.org/example v0.0.0-20200506145455-3957be37c821
# gioui.org/example/kitchen
gioui.org/app/internal/window(.data.rel): unexpected R_X86_64_64 relocation for dynamic symbol wl_surface_interface
gioui.org/app/internal/window(.data.rel): unexpected R_X86_64_64 relocation for dynamic symbol wl_surface_interface
gioui.org/app/internal/window(.data.rel): unexpected R_X86_64_64 relocation for dynamic symbol wl_surface_interface
gioui.org/app/internal/window(.data.rel): unhandled relocation for wl_surface_interface (type 45 (SDYNIMPORT) rtype 1 (R_ADDR))
gioui.org/app/internal/window(.data.rel): unhandled relocation for wl_surface_interface (type 45 (SDYNIMPORT) rtype 1 (R_ADDR))
gioui.org/app/internal/window(.data.rel): relocation target wl_seat_interface not defined
gioui.org/app/internal/window(.data.rel): unhandled relocation for wl_surface_interface (type 45 (SDYNIMPORT) rtype 1 (R_ADDR))
gioui.org/app/internal/window(.data.rel): relocation target wl_seat_interface not defined
gioui.org/app/internal/window(.data.rel): relocation target wl_output_interface not defined
gioui.org/app/internal/window._Cvar_wl_compositor_interface: relocation target wl_compositor_interface not defined
gioui.org/app/internal/window._Cvar_wl_output_interface: relocation target wl_output_interface not defined
gioui.org/app/internal/window._Cvar_wl_seat_interface: relocation target wl_seat_interface not defined
gioui.org/app/internal/window._Cvar_wl_shm_interface: relocation target wl_shm_interface not defined

darwin/amd64 with

diff --git a/src/cmd/link/internal/amd64/asm.go b/src/cmd/link/internal/amd64/asm.go
index c2d54703c1..8ee95b66e7 100644
--- a/src/cmd/link/internal/amd64/asm.go
+++ b/src/cmd/link/internal/amd64/asm.go
@@ -181,7 +181,7 @@ func adddynrel2(target *ld.Target, ldr *loader.Loader, syms *ld.ArchSyms, s load
                su.SetRelocType(rIdx, objabi.R_ADDR)

                if targType == sym.SDYNIMPORT {
-                       ldr.Errorf(s, "unexpected reloc for dynamic symbol %s", ldr.SymName(targ))
+                       ldr.Errorf(s, "unexpected reloc %s for dynamic symbol %s", r.Type(), ldr.SymName(targ))
                }
                return true

applied:

$ go run -ldflags=-linkmode=internal gioui.org/example/kitchen
# gioui.org/example/kitchen
gioui.org/app/headless(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSOpenGLPixelFormat
gioui.org/app/headless(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSOpenGLContext
gioui.org/app/internal/window(__DATA/__objc_data): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSOpenGLView
gioui.org/app/internal/window(__DATA/__objc_data): unexpected reloc RelocType(2048) for dynamic symbol _objc_empty_cache
gioui.org/app/internal/window(__DATA/__objc_data): unexpected reloc RelocType(2048) for dynamic symbol OBJC_METACLASS_$_NSObject
gioui.org/app/internal/window(__DATA/__objc_data): unexpected reloc RelocType(2048) for dynamic symbol OBJC_METACLASS_$_NSOpenGLView
gioui.org/app/internal/window(__DATA/__objc_data): unexpected reloc RelocType(2048) for dynamic symbol _objc_empty_cache
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSArray
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSOpenGLPixelFormat
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSOpenGLContext
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSEvent
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSRunningApplication
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSApplication
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSMenuItem
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSMenu
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSWindow
gioui.org/app/internal/window(__DATA/__objc_classrefs): unexpected reloc RelocType(2048) for dynamic symbol OBJC_CLASS_$_NSString
gioui.org/app/internal/window(__DATA/__cfstring): unexpected reloc RelocType(2048) for dynamic symbol __CFConstantStringClassReference
gioui.org/app/internal/window(__DATA/__cfstring): unexpected reloc RelocType(2048) for dynamic symbol __CFConstantStringClassReference
gioui.org/app/internal/window(__DATA/__cfstring): unexpected reloc RelocType(2048) for dynamic symbol __CFConstantStringClassReference
gioui.org/app/internal/window(__DATA/__cfstring): unexpected reloc RelocType(2048) for dynamic symbol __CFConstantStringClassReference
/Users/elias/go-tip/pkg/tool/darwin_amd64/link: too many errors

android/*, buildmode=c-shared:

$ GOFLAGS= go run gioui.org/cmd/gogio -target android -linkmode internal gioui.org/example/kitchen
gogio: go build -ldflags=-w -s -X gioui.org/app/internal/log.appID=org.gioui.kitchen -linkmode=internal -buildmode=c-shared -o /tmp/gogio-146873352/jni/x86_64/libgio.so gioui.org/example/kitchen failed: # gioui.org/example/kitchen
/home/elias/dev/go-tip/pkg/tool/linux_amd64/link: internal linking requested but external linking required: android/amd64 requires external linking

With

diff --git src/cmd/internal/sys/supported.go src/cmd/internal/sys/supported.go
index c27b3b986d..72f6707b36 100644
--- src/cmd/internal/sys/supported.go
+++ src/cmd/internal/sys/supported.go
@@ -35,9 +35,9 @@ func MSanSupported(goos, goarch string) bool {
 func MustLinkExternal(goos, goarch string) bool {
        switch goos {
        case "android":
-               if goarch != "arm64" {
-                       return true
-               }
+               //if goarch != "arm64" {
+               //return true
+               //}
        case "darwin":
                if goarch == "arm64" {
                        return true
diff --git src/cmd/link/internal/ld/config.go src/cmd/link/internal/ld/config.go
index 2373b500e3..31b5b94f3a 100644
--- src/cmd/link/internal/ld/config.go
+++ src/cmd/link/internal/ld/config.go
@@ -189,9 +189,9 @@ func mustLinkExternal(ctxt *Link) (res bool, reason string) {
        if iscgo && ctxt.Arch.InFamily(sys.MIPS64, sys.MIPS, sys.PPC64) {
                return true, objabi.GOARCH + " does not support internal cgo"
        }
-       if iscgo && objabi.GOOS == "android" {
+       /*if iscgo && objabi.GOOS == "android" {
                return true, objabi.GOOS + " does not support internal cgo"
-       }
+       }*/
 
        // When the race flag is set, the LLVM tsan relocatable file is linked
        // into the final binary, which means external linking is required because
@@ -205,7 +205,7 @@ func mustLinkExternal(ctxt *Link) (res bool, reason string) {
        case BuildModeCArchive:
                return true, "buildmode=c-archive"
        case BuildModeCShared:
-               return true, "buildmode=c-shared"
+               //return true, "buildmode=c-shared"
        case BuildModePIE:
                switch objabi.GOOS + "/" + objabi.GOARCH {
                case "linux/amd64", "linux/arm64", "android/arm64":

the gogio -target android build completes, but the resulting apk fails at runtime with:

java.lang.UnsatisfiedLinkError: dlopen failed: "/data/app/org.gioui.kitchen-xs1nOo55BAjSrffNWB8gBQ==/lib/x86_64/libgio.so" has unexpected e_type: 2

(e_type 2 is ET_EXEC, the loader expects ET_DYN)

darwin/arm64 (iOS):

$ go run gioui.org/cmd/gogio -target ios -linkmode internal gioui.org/example/kitchen
gogio: go build -ldflags=-s -w -X gioui.org/app/internal/log.appID=org.gioui.kitchen -linkmode=internal -buildmode=c-archive -o /var/folders/_7/lnt35k555hl2bs7fjygkhgx00000gp/T/gogio-897284295/gio-arm64 -tags ios gioui.org/example/kitchen failed: # gioui.org/example/kitchen
/Users/elias/go-tip/pkg/tool/darwin_amd64/link: internal linking requested but external linking required: darwin/arm64 requires external linking

With

$ git diff
diff --git a/src/cmd/internal/sys/supported.go b/src/cmd/internal/sys/supported.go
index c27b3b986d..5ffc088358 100644
--- a/src/cmd/internal/sys/supported.go
+++ b/src/cmd/internal/sys/supported.go
@@ -39,9 +39,9 @@ func MustLinkExternal(goos, goarch string) bool {
                        return true
                }
        case "darwin":
-               if goarch == "arm64" {
+               /*if goarch == "arm64" {
                        return true
-               }
+               }*/
        }
        return false
 }
diff --git a/src/cmd/link/internal/ld/config.go b/src/cmd/link/internal/ld/config.go
index 2373b500e3..c972961c4b 100644
--- a/src/cmd/link/internal/ld/config.go
+++ b/src/cmd/link/internal/ld/config.go
@@ -203,7 +203,7 @@ func mustLinkExternal(ctxt *Link) (res bool, reason string) {
        // Some build modes require work the internal linker cannot do (yet).
        switch ctxt.BuildMode {
        case BuildModeCArchive:
-               return true, "buildmode=c-archive"
+               //return true, "buildmode=c-archive"
        case BuildModeCShared:
                return true, "buildmode=c-shared"
        case BuildModePIE:
$ go run gioui.org/cmd/gogio -target ios -linkmode internal gioui.org/example/kitchen
gogio: go build -ldflags=-s -w -X gioui.org/app/internal/log.appID=org.gioui.kitchen -linkmode=internal -buildmode=c-archive -o /var/folders/_7/lnt35k555hl2bs7fjygkhgx00000gp/T/gogio-032492589/gio-arm64 -tags ios gioui.org/example/kitchen failed: # gioui.org/example/kitchen
loadmacho: /Users/elias/Library/Caches/go-build/b5/b5e636c59e2b0dce3881e4c1dfe5f49890fd7ffc8dabfa89e4c44faee895a24e-d(_x001.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/b5/b5e636c59e2b0dce3881e4c1dfe5f49890fd7ffc8dabfa89e4c44faee895a24e-d(_x002.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/b5/b5e636c59e2b0dce3881e4c1dfe5f49890fd7ffc8dabfa89e4c44faee895a24e-d(_x003.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/a1/a1b18baa12e8cfcff9d2149878989bf8d3212abe425e72aab827c249f9c1f9aa-d(_x001.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/a1/a1b18baa12e8cfcff9d2149878989bf8d3212abe425e72aab827c249f9c1f9aa-d(_x002.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/a1/a1b18baa12e8cfcff9d2149878989bf8d3212abe425e72aab827c249f9c1f9aa-d(_x003.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/a1/a1b18baa12e8cfcff9d2149878989bf8d3212abe425e72aab827c249f9c1f9aa-d(_x004.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/a1/a1b18baa12e8cfcff9d2149878989bf8d3212abe425e72aab827c249f9c1f9aa-d(_x005.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/a1/a1b18baa12e8cfcff9d2149878989bf8d3212abe425e72aab827c249f9c1f9aa-d(_x006.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/bf/bfc3cd3cbc11f70eee4813613929853ddba3f7efa3b8e6c59544b47c17e10f16-d(_x001.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/bf/bfc3cd3cbc11f70eee4813613929853ddba3f7efa3b8e6c59544b47c17e10f16-d(_x002.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/bf/bfc3cd3cbc11f70eee4813613929853ddba3f7efa3b8e6c59544b47c17e10f16-d(_x003.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x001.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x002.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x003.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x004.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x005.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x006.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x007.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x008.o): mach-o arm64 unimplemented
loadmacho: /Users/elias/Library/Caches/go-build/6f/6f1a0820507421e19028b78d9efe163c52175627d10178d0968d09fd3d95379d-d(_x009.o): mach-o arm64 unimplemented
/Users/elias/go-tip/pkg/tool/darwin_amd64/link: too many errors

What did you expect to see?

A running program.

What did you see instead?

Link errors.

By itself, -linkmode=internal doesn't seem important when using Cgo, but in combination with #38917 internal linking is a crucial step towards dropping the native toolchain altogether.

@ianlancetaylor
Copy link
Contributor

Copying from my comment on #38917 :

Some cgo code relies on libraries that are written in C++. C++ code requires extra work from the linker to support things like global constructors and destructors and exception handling. If we can't invoke the external linker, we have to implement all of that code in cmd/link, and we have to maintain it as C++ changes.

This can be done, of course. I've done it myself in projects like the gold linker. But it's hard.

@cherrymui
Copy link
Member

We might be able to handle this specific case of R_X86_64_64. Instead of supporting the full C/C++ linking, if you have a list of relocations that you need support, and not involving C++, we may be able to do that.

@ianlancetaylor
Copy link
Contributor

That is a good point: we can likely do a better job of supporting pure C libraries that don't use __attribute__((constructor)).

@eliasnaur
Copy link
Contributor Author

The interesting targets for me are iOS, macOS, Android, and to a lesser extent Linux. I've updated the original report with the errors reported when internal linking is forced. Thank you.

@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 11, 2020
@toothrot toothrot added this to the Backlog milestone May 11, 2020
theclapp pushed a commit to theclapp/gio-mirror that referenced this issue May 13, 2020
For experimenting with golang/go#38918.

Signed-off-by: Elias Naur <mail@eliasnaur.com>
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Status: Triage Backlog
Development

No branches or pull requests

6 participants