New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: OS X codesign only works with -ldflags -s #11887

Open
derekparker opened this Issue Jul 27, 2015 · 20 comments

Comments

Projects
None yet
9 participants
@derekparker
Contributor

derekparker commented Jul 27, 2015

Using the Go 1.5 compiler toolchain and compiling a binary that uses CGO and then codesigning that binary, upon attempting to execute the program it will be killed immediately by the system. Codesigning a non-cgo binary works just fine, as expected.

Given the following small program:

package main

/*
int
cfunc() {
    return 2;
}
*/
import "C"
import "fmt"

func main() {
    fmt.Println("cfunc says:", C.cfunc())
}

and building simply with go build and running codesign -f -s my_cert ./myprog, when attempting to execute the program in a shell the response is "killed: ./myprog".

The system log reports:

$ taskgated[92]: no signature for pid=1991 (cannot make code: UNIX[No such process])

Tested against Go1.5beta2 and:

$ go version devel +765cea2 Mon Jul 27 18:03:45 2015 +0000 darwin/amd64

@ianlancetaylor ianlancetaylor added this to the Go1.5 milestone Jul 27, 2015

@ianlancetaylor ianlancetaylor changed the title from OSX/cgo codesign issue to cmd/link: codesign of cgo binary produces program that can not be executed Jul 27, 2015

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jul 27, 2015

Please run "go build -ldflags=-v" and attach the output. I'd like to confirm that the linker is invoking the external linker. Thanks.

@derekparker

This comment has been minimized.

Contributor

derekparker commented Jul 27, 2015


$ go build -ldflags=-v

# github.com/derekparker/testcgo
HEADER = -H1 -T0x2000 -D0x0 -R0x1000
searching for runtime.a in $WORK/runtime.a
searching for runtime.a in /usr/local/go/pkg/darwin_amd64/runtime.a
 0.00 deadcode
 0.02 pclntab=334118 bytes, funcdata total 69364 bytes
 0.02 dodata
 0.02 symsize = 0
 0.03 symsize = 0
 0.03 reloc
 0.04 reloc
 0.05 asmb
 0.05 codeblk
 0.07 datblk
 0.07 dwarf
 0.08 symsize = 0
 0.13 dwarf pass 2.
 0.14 sym
 0.16 headr
host link: "clang" "-m64" "-gdwarf-2" "-Wl,-no_pie,-headerpad,1144" "-Wl,-pagezero_size,4000000" "-o" "/var/folders/qk/sdqgc3hn4v7_zxpkbmms53k00000gn/T/go-build627095567/github.com/derekparker/testcgo/_obj/exe/a.out" "-Qunused-arguments" "/var/folders/qk/sdqgc3hn4v7_zxpkbmms53k00000gn/T/go-link-928391324/000000.o" "/var/folders/qk/sdqgc3hn4v7_zxpkbmms53k00000gn/T/go-link-928391324/000001.o" "/var/folders/qk/sdqgc3hn4v7_zxpkbmms53k00000gn/T/go-link-928391324/go.o" "-g" "-O2" "-g" "-O2" "-lpthread"
 0.26 cpu time
42397 symbols
30720 liveness data
@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jul 27, 2015

Thanks. We apparently have a program built by clang for which codesign does not work. Off hand I have no guess as to what the Go linker might be doing to cause this to fail. This will have to be tackled by somebody who understands how codesign works on Darwin.

@rsc

This comment has been minimized.

Contributor

rsc commented Jul 28, 2015

If you build with -ldflags -s, which drops the DWARF information from the executable, then codesign produces a valid binary. I think this is mostly not Go's fault (except that Go works hard to put DWARF info in the binaries, unlike most OS X programs). Certainly for Go 1.5 but possibly long term we can suggest that people who need codesign can use -ldflags -s.

@rsc rsc changed the title from cmd/link: codesign of cgo binary produces program that can not be executed to cmd/link: OS X codesign only works with -ldflags -s Jul 28, 2015

@rsc rsc modified the milestones: Unplanned, Go1.5 Jul 28, 2015

@derekparker

This comment has been minimized.

Contributor

derekparker commented Jul 28, 2015

Thanks, that does seem to work.

derekparker added a commit to derekparker/delve that referenced this issue Jul 28, 2015

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Jul 29, 2015

Any thoughts on whether we should document this for Go 1.5, and where we should do it? Ideally we should put the docs in the place where someone who encounters the problem will look, but where would that be?

derekparker referenced this issue in derekparker/delve Aug 3, 2015

Conditionally set ld flags in Makefile for Go version
Passing the '-s' flag to the linker in < Go 1.5 emits an error and
produces a binary that, once codesigned (I believe) will cause an
immediate bus error and terminate.

@zimbatm zimbatm referenced this issue Aug 30, 2015

Closed

go: 1.5 darwin #9558

@stuartcarnie

This comment has been minimized.

stuartcarnie commented Dec 3, 2015

-ldflags -s no longer seems to work with go 1.5.2 and results in same error in logs

12/3/15 1:32:57.505 PM taskgated[59690]: no signature for pid=59709 (cannot make code: UNIX[No such process])
@fasterthanlime

This comment has been minimized.

fasterthanlime commented Mar 22, 2016

Issue still exists on 1.6, ldflags -s doesn't work either:

ld: warning: option -s is obsolete and being ignored
ld: internal error: atom not found in symbolIndex(__ZN6brotli10FindBlocksIhLi256EEEvPKT_mdRKNSt3__16vectorINS_9HistogramIXT0_EEENS4_9allocatorIS7_EEEEPh) for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Probably going to forgo codesigning for now, but eager to learn about another workaround or a real fix! 🌟

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 22, 2016

The ld: warning: option -s is obsolete... suggests you are somehow passing the -s to the host linker instead of to cmd/link.

@fasterthanlime

This comment has been minimized.

fasterthanlime commented Mar 22, 2016

The ld: warning: option -s is obsolete... suggests you are somehow passing the -s to the host linker instead of to cmd/link.

That would be really hard to do accidentally, one would have to run:

go build -ldflags "-extldflags -s"

instead of:

go build -ldflags "-s"

and I can assure you I was doing the latter. However, I suspect cmd/link is passing -s down to clang.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 22, 2016

If the problem with cmd/link is just dwarf, then try -ldflags -w instead of
-ldflags -s.

On Tue, Mar 22, 2016 at 12:00 PM Amos Wenger notifications@github.com
wrote:

The ld: warning: option -s is obsolete... suggests you are somehow passing
the -s to the host linker instead of to cmd/link.

That would be really hard to do accidentally, one would have to run:

go build -ldflags "-extldflags -s"

instead of:

go build -ldflags "-s"

and I can assure you I was doing the latter. However, I suspect cmd/link
is passing -s down to clang.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#11887 (comment)

@fasterthanlime

This comment has been minimized.

fasterthanlime commented Mar 22, 2016

If the problem with cmd/link is just dwarf, then try -ldflags -w instead of
-ldflags -s.

With -w, the compilation & linking succeeds, a working binary is produced, but codesign-ing that binary breaks it, as in the original report.

@gazed

This comment has been minimized.

gazed commented Mar 23, 2016

I'm finding that ldflags -s does work with go 1.6 and codesign
based on the following test code in 11887.go

package main
import "C" // Need this line to break codesign without "ldflags -s".
func main() {
    println("Hello OSX.")
}
: go build -ldflags -s -o 11887s 11887.go
: go build -o 11887n 11887.go
: codesign -s "3rd Party Mac Developer Application: Galvanized Logic Inc." 11887s
: codesign -s "3rd Party Mac Developer Application: Galvanized Logic Inc." 11887n
: ./11887s
Hello OSX.
: ./11887n
Killed: 9
: go version
go version go1.6 darwin/amd64
@fasterthanlime

This comment has been minimized.

fasterthanlime commented Mar 23, 2016

I'm finding that ldflags -s does work with go 1.6 and codesign
based on the following test code in 11887.go

Following the exact same protocol, but with my program, I'm able to reproduce your results on my local MacbookPro, but not on my build server.

$ go build -ldflags -s -o butler-s .
# github.com/itchio/butler
/usr/local/Cellar/go/1.6/libexec/pkg/tool/darwin_amd64/link: running clang++ failed: exit status 1
ld: warning: option -s is obsolete and being ignored
ld: internal error: atom not found in symbolIndex(__ZN6brotli10FindBlocksIhLi256EEEvPKT_mdRKNSt3__16vectorINS_9HistogramIXT0_EEENS4_9allocatorIS7_EEEEPh) for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The build server has an older version of clang & OSX:

$ clang --version
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin14.5.0
Thread model: posix

$ sw_vers -productVersion
10.10.5

My local computer has the latest stable of both:

$ clang --version
Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

$ sw_vers -productVersion
10.11.3

Note that, even on my local machine, cmd/link does pass -s down to clang, which prints a warning — it just looks like cmd/link swallows linker warnings if the link step is successful:

$ go build -v -x -ldflags "-s -v" -o butler-s .
# (manually suppressed output for readability)
# note the "-s" below
host link: "clang++" "-m64" "-s" "-Wl,-no_pie,-headerpad,1144" "-Wl,-pagezero_size,4000000" "-o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-build649125102/github.com/itchio/butler/_obj/exe/a.out" "-Qunused-arguments" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/go.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000000.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000001.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000002.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000003.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000004.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000005.o" "/var/folders/kj/vbzcvcf94hn0hkxpc6d97zcc0000gn/T/go-link-896862525/000006.o" "-g" "-O2" "-g" "-O2" "-g" "-O2" "-lpthread" "-g" "-O2" "-framework" "CoreFoundation" "-framework" "Security" "-g" "-O2" "-g" "-O2" "-g" "-O2"
# here's the warning
ld: warning: option -s is obsolete and being ignored

I'd understand if this bug doesn't receive a lot of attention, since recent clangs seem to handle it just fine. I'll just upgrade my build server, thanks for the help! 🌟

edit: if anyone's interested in reproducing the above, the program I'm building is https://github.com/itchio/butler

edit 2: note that the simple 11887.go test case posted by @gazed above isn't enough to reproduce the problem

@kujenga

This comment has been minimized.

kujenga commented Mar 28, 2017

I'm currently running into this exact issue without performing any explicit code signing.

I updated my mac yesterday with the latest command line tools and Xcode, version 8.3, and I believe exactly then is when I started seeing this issue. I'm on macOS version 10.12.4 (16E195).

The failure happens even when doing a vanilla go run on a single file containing Cgo code. I created a repository with a minimal example of the failure here: https://github.com/kujenga/brokencgo

Here's a file causing failures when I go run it:

package main

import (
	"unsafe"
)

/*
#include <stdio.h>
#include <stdlib.h>

void myprint(char* s) {
	printf("%s\n", s);
}
*/
import "C"

func main() {
	cs := C.CString("Hello from stdio\n")
	C.myprint(cs)
	C.free(unsafe.Pointer(cs))
}

My theory is that something in the update to the command line tools is causing a breakage here.

@kujenga

This comment has been minimized.

kujenga commented Mar 28, 2017

It looks like the new issues are now being tracked here: #19734

@gopherbot

This comment has been minimized.

gopherbot commented Mar 30, 2017

CL https://golang.org/cl/38853 mentions this issue.

@rsc

This comment has been minimized.

Contributor

rsc commented Mar 30, 2017

I see now what happened with -s above: passing -s to the Go linker relays it to the macOS linker. And even though the macOS linker says it is ignoring -s, it seems to not be ignoring it quite enough. This was #19775 just now.

I expect the fix for that will be in Go 1.8.1, so that you can use -ldflags=-s successfully on binaries with external linking. I hope that will get codesign working again.

In fact codesign might start working out of the box without any special flags. At least on my machine right now:

$ cd go/src/runtime/testdata/testprogcgo
$ go build
$ codesign -f -s - testprogcgo
$ ./testprogcgo
usage: ./testprogcgo name-of-test
$ 

I don't know if using -s - instead of a real identity bypasses the problems others have seen. If anyone else who can reproduce a problem can check that codesign -f -s - yourbinary fails as well, that'd be great.

Thanks.

gopherbot pushed a commit that referenced this issue Mar 30, 2017

cmd/link: disable mach-o dwarf munging with -w (in addition to -s)
Might as well provide a way around the mach-o munging
that doesn't require stripping all symbols.
After all, -w does mean no DWARF.

For #11887, #19734, and anyone else that needs to disable
this code path without losing the symbol table.

Change-Id: I254b7539f97fb9211fa90f446264b383e7f3980f
Reviewed-on: https://go-review.googlesource.com/38853
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Apr 5, 2017

CL https://golang.org/cl/39602 mentions this issue.

gopherbot pushed a commit that referenced this issue Apr 5, 2017

[release-branch.go1.8] cmd/link: disable mach-o dwarf munging with -w…
… (in addition to -s)

Might as well provide a way around the mach-o munging
that doesn't require stripping all symbols.
After all, -w does mean no DWARF.

For #11887, #19734, and anyone else that needs to disable
this code path without losing the symbol table.

Change-Id: I254b7539f97fb9211fa90f446264b383e7f3980f
Reviewed-on: https://go-review.googlesource.com/39602
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>

@xert xert referenced this issue Apr 7, 2017

Closed

signal: killed #64

lparth added a commit to lparth/go that referenced this issue Apr 13, 2017

cmd/link: disable mach-o dwarf munging with -w (in addition to -s)
Might as well provide a way around the mach-o munging
that doesn't require stripping all symbols.
After all, -w does mean no DWARF.

For golang#11887, golang#19734, and anyone else that needs to disable
this code path without losing the symbol table.

Change-Id: I254b7539f97fb9211fa90f446264b383e7f3980f
Reviewed-on: https://go-review.googlesource.com/38853
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@OSMeteor

This comment has been minimized.

OSMeteor commented Apr 24, 2017

use go1.8.1 fixed it good

benma added a commit to benma/bitbox-wallet-app that referenced this issue Nov 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment