Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: macOS on arm64 requires codesigning #42684

Closed
FiloSottile opened this issue Nov 18, 2020 · 119 comments
Closed

cmd/go: macOS on arm64 requires codesigning #42684

FiloSottile opened this issue Nov 18, 2020 · 119 comments

Comments

@FiloSottile
Copy link
Contributor

@FiloSottile FiloSottile commented Nov 18, 2020

On the production Apple Silicon machines, Go binaries are killed at start. #38485 (comment)

It looks like all binaries need to be codesigned now, and indeed running codesign -s - on them lets them run correctly.

This stops go test and go run from working, and requires an extra step after go build to get a functional binary.

This also affects the bootstrapped compiler itself.

@FiloSottile
Copy link
Contributor Author

@FiloSottile FiloSottile commented Nov 18, 2020

@cherrymui indeed, binaries produced by the machine's clang are codesigned, in what looks like the same exact way as the output of codesign -s -.

filippo@Filippos-MacBook-Pro tmp % clang hello.c
filippo@Filippos-MacBook-Pro tmp % ./a.out
Hello M1!
filippo@Filippos-MacBook-Pro tmp % codesign -d -v a.out                                 
Executable=/Users/filippo/tmp/a.out
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=510 flags=0x20002(adhoc,linker-signed) hashes=13+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

Loading

@FiloSottile
Copy link
Contributor Author

@FiloSottile FiloSottile commented Nov 18, 2020

Actually, codesign outputs don't have the linker-signed flag.

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 18, 2020

Thanks, @FiloSottile !

Interesting. The C compiler on the DTK doesn't do that...

Could you try one more thing: run clang -v hello.c to see if the C compiler invokes codesign or how it signs the binary? Thanks!

Loading

@FiloSottile
Copy link
Contributor Author

@FiloSottile FiloSottile commented Nov 18, 2020

It doesn't look like the linker invocation has anything special, so I assume ld just does that (and @jedisct1 said so on Twitter as well https://twitter.com/jedisct1/status/1328862207715794946).

"/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch arm64 -platform_version macos 11.0.0 11.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o a.out -L/usr/local/lib /var/folders/jh/3ydm4lxd71s2__g_x4hny6r00000gn/T/hello-580ca7.o -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.osx.a

I expect external linking should generate codesigned binaries, but trying `go build -ldflags="-linkmode=external" fails immediately with

# tmp
loadinternal: cannot find runtime/cgo

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 18, 2020

loadinternal: cannot find runtime/cgo

This is not a failure. It prints a message but it should successfully link the binary.

Loading

@FiloSottile
Copy link
Contributor Author

@FiloSottile FiloSottile commented Nov 18, 2020

You're right, I do get the binary. Interestingly, it comes out linker-signed but doesn't run, and codesign doesn't work on it.

filippo@Filippos-MacBook-Pro tmp % codesign -d -v tmp                                                       
Executable=/Users/filippo/tmp/tmp
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=7038 flags=0x20002(adhoc,linker-signed) hashes=217+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
filippo@Filippos-MacBook-Pro tmp % ./tmp             
zsh: killed     ./tmp
filippo@Filippos-MacBook-Pro tmp % codesign -s - tmp 
tmp: the codesign_allocate helper tool cannot be found or used

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 18, 2020

Thanks!

Interesting... I'll see if it is possible to generate LC_CODE_SIGNATURE in the linker. If that's not possible, maybe shell out codesign.

Any down side for that? If the user wants to sign with a different identity, would that still be possible?

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 18, 2020

I don't understand this. What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

Loading

@FiloSottile
Copy link
Contributor Author

@FiloSottile FiloSottile commented Nov 18, 2020

Any down side for that?

I guess the main problem would be with cross-compilation, but anyway I guess adhoc signatures only work on the same machine where they are generated.

This would be a major pain for projects distributing binaries. I wonder how Homebrew deals with it, since by default they install pre-built "Bottles".

What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

I don't know, but I can imagine 1) to normalize the execution path such that it always check signatures unconditionally, simplifying it and 2) to block binaries generated on other machines (and not properly signed by a trusted Developer ID) which before was sort of the role of the quarantine attributes.

Loading

@terinjokes
Copy link
Contributor

@terinjokes terinjokes commented Nov 18, 2020

You're right, I do get the binary. Interestingly, it comes out linker-signed but doesn't run, and codesign doesn't work on it.

My understanding is this is a bug already reported to Apple, see Homebrew/brew#9082 (comment)

I wonder how Homebrew deals with it, since by default they install pre-built "Bottles".

Per Homebrew/brew#7857 (comment) I don't think they're close to code signing bottles on ARM yet.

Loading

@rolandshoemaker
Copy link
Member

@rolandshoemaker rolandshoemaker commented Nov 18, 2020

  1. to block binaries generated on other machines (and not properly signed by a trusted Developer ID) which before was sort of the role of the quarantine attributes.

Looks like binaries aren't required to have a signature linked to a Developer ID if they were signed elsewhere. I was able to apply an ad-hoc signature to a binary on my Intel mac and transfer it to my M1 mac and it ran just fine. Appears like binaries just need a signature, doesn't really seem to matter where it came from.

Loading

@fxcoudert
Copy link

@fxcoudert fxcoudert commented Nov 18, 2020

Homebrew maintainer here:

the codesign_allocate helper tool cannot be found or used

That's indeed a bug in codesign, and Apple is aware

My understanding is this is a bug already reported to Apple, see Homebrew/brew#9082 (comment)

Yes. If this occurs, you need to change that file's inode (so copying it someplace and back will work). Once you've done that, you can sign and it will work. See Homebrew/brew#9102 for details.

I don't think they're close to code signing bottles on ARM yet

We do not distribute bottles yet, because our CI is not yet fully operational, but our codebase is otherwise ready. We apply ad-hoc signatures as part of formula installation: Homebrew/brew#9102

Loading

@fxcoudert
Copy link

@fxcoudert fxcoudert commented Nov 18, 2020

As for go, if Go uses its own linker (or anything other than Apple's ld) then it definitely needs to call codesign -s - on the binaries it produces: executables, shared libraries, etc.

Loading

@mislav
Copy link

@mislav mislav commented Nov 18, 2020

@FiloSottile (and others): thank you for the information! 🙇

Do you know if it is possible to use darwin/amd64 architecture to cross-compile binaries that are able to execute on darwin/arm64, providing that the binaries are explicitly signed?

I am interested in what are the options to distribute binaries for Go projects that work on Apple Silicon without having to actually compile them on Apple Silicon.

Loading

@bwesterb
Copy link

@bwesterb bwesterb commented Nov 18, 2020

@mislav Cross-compiling for darwin/arm64 works fine from darwin/amd64 and even linux/amd64. Obviously the resulting binaries are not signed. They will run if signed ad hoc on the same machine they will run. Binaries ad hoc signed on one machine didn't work on another machine for me as well.

Loading

@jameshartig
Copy link
Contributor

@jameshartig jameshartig commented Nov 18, 2020

Binaries ad hoc signed on one machine didn't work on another machine for me.

This comment seems to disagree: #42684 (comment)

I was able to apply an ad-hoc signature to a binary on my Intel mac and transfer it to my M1 mac and it ran just fine.

Loading

@bwesterb
Copy link

@bwesterb bwesterb commented Nov 18, 2020

Ok, I tested it again and now cross-signing does work. I might've transferred the wrong file earlier. Whoops.

Loading

@FiloSottile
Copy link
Contributor Author

@FiloSottile FiloSottile commented Nov 18, 2020

Codesigning from cmd/link should be possible. https://github.com/isignpy/isign by @neilk does that for iOS.

Loading

@networkimprov
Copy link

@networkimprov networkimprov commented Nov 18, 2020

The Zig project is also working on this here: ziglang/zig#7103 (thanks @komuw).

Loading

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 18, 2020

Thanks for the pointers. I'll look into them.

Loading

@fxcoudert
Copy link

@fxcoudert fxcoudert commented Nov 18, 2020

Binaries ad hoc signed on one machine didn't work on another machine for me

I can confirm that a linker or other ad hoc signature is sufficient to run the binary (or shared library) on any machine.

Loading

@aclements aclements added this to the Go1.16 milestone Nov 18, 2020
gopherbot pushed a commit that referenced this issue Dec 1, 2020
On macOS/ARM64, the kernel requires that binaries must have a
valid code signature to run. The C toolchain code-signs the
binary at link time. We do the same.

It is more subtle for Go because we stamp the buildid after
linking. As the signature contains hashes of the entire file
(except the signature itself), we must (re)generate the signature
after stamping the buildid.

This CL adds a new codesign package, which provides
functionality to generate the code signature. It is a separate
internal package so it can be used both in the linker and by the
go command. The next CLs will add code-signing to the linker and
the go command.

Updates #38485, #42684.

Change-Id: Id46801a6665beebaab0eb413ff2e64c5b9467059
Reviewed-on: https://go-review.googlesource.com/c/go/+/272254
Trust: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
gopherbot pushed a commit that referenced this issue Dec 1, 2020
The code signature contains hashes of the entire file (except the
signature itself), including the buildid. Therefore, the buildid
cannot depend on the signature. Otherwise updating buildid will
invalidate the signature, and vice versa. As we cannot change the
code-signing algorithm, we can only change buildid calculation.

This CL changes the buildid calculation to exclude the Mach-O
code signature. So updating code signature after stamping the
buildid will not invalidate the buildid.

Updates #38485, #42684.

Change-Id: I8a9e2e25ca9dc00d9556d13b81652f43bbf6a084
Reviewed-on: https://go-review.googlesource.com/c/go/+/272255
Trust: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
gopherbot pushed a commit that referenced this issue Dec 1, 2020
This CL lets the linker code-sign output binaries on
darwin/arm64, as the kernel requires binaries must be signed in
order to run.

This signature will likely be invalidated when we stamp the
buildid after linking. We still do it in the linker, for
- plain "go tool link" works.
- the linker generates the LC_CODE_SIGNATURE load command with
  the right size and offset, so we don't need to update it when
  stamping the buildid.

Updates #38485, #42684.

Change-Id: Ia306328906d73217221ba31093fe61a935a46122
Reviewed-on: https://go-review.googlesource.com/c/go/+/272256
Trust: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
gopherbot pushed a commit that referenced this issue Dec 1, 2020
…ldid

As the code signature contains hashes of the entire file (except
the signature itself), rewriting buildid will invalidate the
signature. This CL makes it regenerate the signature when
rewriting the buildid. It only does it when the file already has
a code signature, with proper size (darwin/arm64 binaries
generated by the Go linker should have).

Updates #38485, #42684.

Change-Id: I082d9e5808b0ee6a35f9c362d7262aadd9113c81
Reviewed-on: https://go-review.googlesource.com/c/go/+/272257
Trust: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
@gopherbot gopherbot closed this in cf7aa58 Dec 1, 2020
briansmith added a commit to briansmith/rustls that referenced this issue Dec 2, 2020
Fix the "build bogo_shim if it doesn't exist" logic; it was broken even
on Linux as newer versions of Rust don't allow `--features` to be used
at workspace level.

I had to bootstrap Go on a Linux machine using the procedure at
golang/go#42684 (comment):
```
git clone https://go.googlesource.com/go
cd go
git fetch https://go.googlesource.com/go refs/changes/58/272258/1 && git checkout FETCH_HEAD
cd src
GOOS=darwin GOARCH=arm64 ./bootstrap.bash
```

This version of Go required me to have a go.mod file. Change the way Bogo is
downloaded from BoringSSL's repo to get this to work. The script tries to use
sparse checkouts and a single-revision checkout but it is still slow so maybe
one or both of those things aren't working as I expect.

In particular, avoid `wget` since my Mac doesn't have `wget` available.

Remove two of the patches to Bogo that seem to not be necessary if using a
newer version of Go. Patch the remaining patch to work with the new directory
structure.

I verified that ./runme runs the tests and spits out "PASS" at the end on both macOS
and Linux.
briansmith added a commit to briansmith/rustls that referenced this issue Dec 2, 2020
Fix the "build bogo_shim if it doesn't exist" logic; it was broken even
on Linux as newer versions of Rust don't allow `--features` to be used
at workspace level.

I had to bootstrap Go on a Linux machine using the procedure at
golang/go#42684 (comment):
```
git clone https://go.googlesource.com/go
cd go
git fetch https://go.googlesource.com/go refs/changes/58/272258/1 && git checkout FETCH_HEAD
cd src
GOOS=darwin GOARCH=arm64 ./bootstrap.bash
```

This version of Go required me to have a go.mod file. Change the way Bogo is
downloaded from BoringSSL's repo to get this to work. The script now uses a
sparse checkout with depth 1, which seems pretty fast. In particular, avoid
`wget` since my Mac doesn't have `wget` available.

Remove two of the patches to Bogo that seem to not be necessary if using a
newer version of Go. Patch the remaining patch to work with the new directory
structure.

I verified that ./runme runs the tests and spits out "PASS" at the end on both macOS
and Linux.
ctz added a commit to rustls/rustls that referenced this issue Dec 5, 2020
Fix the "build bogo_shim if it doesn't exist" logic; it was broken even
on Linux as newer versions of Rust don't allow `--features` to be used
at workspace level.

I had to bootstrap Go on a Linux machine using the procedure at
golang/go#42684 (comment):
```
git clone https://go.googlesource.com/go
cd go
git fetch https://go.googlesource.com/go refs/changes/58/272258/1 && git checkout FETCH_HEAD
cd src
GOOS=darwin GOARCH=arm64 ./bootstrap.bash
```

This version of Go required me to have a go.mod file. Change the way Bogo is
downloaded from BoringSSL's repo to get this to work. The script now uses a
sparse checkout with depth 1, which seems pretty fast. In particular, avoid
`wget` since my Mac doesn't have `wget` available.

Remove two of the patches to Bogo that seem to not be necessary if using a
newer version of Go. Patch the remaining patch to work with the new directory
structure.

I verified that ./runme runs the tests and spits out "PASS" at the end on both macOS
and Linux.
edigaryev added a commit to cirruslabs/cirrus-cli that referenced this issue Dec 6, 2020
edigaryev added a commit to cirruslabs/cirrus-cli that referenced this issue Dec 7, 2020
@jimtut
Copy link

@jimtut jimtut commented Dec 17, 2020

You can do that all on the m1 mac by replacing the last line with
arch --x86_64 env GODEBUG=asyncpreemptoff=1 GOOS=darwin GOARCH=arm64 ./bootstrap.bash

This doesn't see true. I'm on an M1 Mac mini, and these commands:

git clone https://go.googlesource.com/go
cd go/src
arch --x86_64 env GODEBUG=asyncpreemptoff=1 GOOS=darwin GOARCH=arm64 ./bootstrap.bash

give an error:

ERROR: Cannot find /Users/jim/go1.4/bin/go.
Set $GOROOT_BOOTSTRAP to a working Go tree >= Go 1.4.

UPDATE: Piecing together various bits and bobs above, this hung the first time, but worked the second:

  1. Download go1.15.6.darwin-amd64.pkg from https://golang.org/doc/install
  2. Run the PKG file. It's x86, but Rosetta 2 will run the installer, and place an x86 Go install in /usr/local/go
  3. git clone https://go.googlesource.com/go
  4. cd go/src
  5. arch --x86_64 env GOROOT_BOOTSTRAP=/usr/local/go GODEBUG=asyncpreemptoff=1 GOOS=darwin GOARCH=arm64 ./bootstrap.bash

Loading

@fayep
Copy link

@fayep fayep commented Dec 17, 2020

Loading

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Dec 17, 2020

You can now download a native macOS ARM64 binary release from https://golang.org/dl/#go1.16beta1, as it was released today. Look for go1.16beta1.darwin-arm64.pkg or go1.16beta1.darwin-arm64.tar.gz. Then you can use Go 1.16 Beta 1 itself, or use it to build a newer gotip version.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet