Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: macOS on arm64 requires codesigning #42684

Open
FiloSottile opened this issue Nov 18, 2020 · 116 comments
Open

cmd/go: macOS on arm64 requires codesigning #42684

FiloSottile opened this issue Nov 18, 2020 · 116 comments

Comments

@FiloSottile
Copy link
Member

@FiloSottile FiloSottile commented Nov 18, 2020

On the production Apple Silicon machines, Go binaries are killed at start. #38485 (comment)

It looks like all binaries need to be codesigned now, and indeed running codesign -s - on them lets them run correctly.

This stops go test and go run from working, and requires an extra step after go build to get a functional binary.

This also affects the bootstrapped compiler itself.

@FiloSottile
Copy link
Member Author

@FiloSottile FiloSottile commented Nov 18, 2020

@cherrymui indeed, binaries produced by the machine's clang are codesigned, in what looks like the same exact way as the output of codesign -s -.

filippo@Filippos-MacBook-Pro tmp % clang hello.c
filippo@Filippos-MacBook-Pro tmp % ./a.out
Hello M1!
filippo@Filippos-MacBook-Pro tmp % codesign -d -v a.out                                 
Executable=/Users/filippo/tmp/a.out
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=510 flags=0x20002(adhoc,linker-signed) hashes=13+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
@FiloSottile
Copy link
Member Author

@FiloSottile FiloSottile commented Nov 18, 2020

Actually, codesign outputs don't have the linker-signed flag.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 18, 2020

Thanks, @FiloSottile !

Interesting. The C compiler on the DTK doesn't do that...

Could you try one more thing: run clang -v hello.c to see if the C compiler invokes codesign or how it signs the binary? Thanks!

@FiloSottile
Copy link
Member Author

@FiloSottile FiloSottile commented Nov 18, 2020

It doesn't look like the linker invocation has anything special, so I assume ld just does that (and @jedisct1 said so on Twitter as well https://twitter.com/jedisct1/status/1328862207715794946).

"/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch arm64 -platform_version macos 11.0.0 11.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o a.out -L/usr/local/lib /var/folders/jh/3ydm4lxd71s2__g_x4hny6r00000gn/T/hello-580ca7.o -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.osx.a

I expect external linking should generate codesigned binaries, but trying `go build -ldflags="-linkmode=external" fails immediately with

# tmp
loadinternal: cannot find runtime/cgo
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 18, 2020

loadinternal: cannot find runtime/cgo

This is not a failure. It prints a message but it should successfully link the binary.

@FiloSottile
Copy link
Member Author

@FiloSottile FiloSottile commented Nov 18, 2020

You're right, I do get the binary. Interestingly, it comes out linker-signed but doesn't run, and codesign doesn't work on it.

filippo@Filippos-MacBook-Pro tmp % codesign -d -v tmp                                                       
Executable=/Users/filippo/tmp/tmp
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=7038 flags=0x20002(adhoc,linker-signed) hashes=217+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
filippo@Filippos-MacBook-Pro tmp % ./tmp             
zsh: killed     ./tmp
filippo@Filippos-MacBook-Pro tmp % codesign -s - tmp 
tmp: the codesign_allocate helper tool cannot be found or used
@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 18, 2020

Thanks!

Interesting... I'll see if it is possible to generate LC_CODE_SIGNATURE in the linker. If that's not possible, maybe shell out codesign.

Any down side for that? If the user wants to sign with a different identity, would that still be possible?

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 18, 2020

I don't understand this. What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

@FiloSottile
Copy link
Member Author

@FiloSottile FiloSottile commented Nov 18, 2020

Any down side for that?

I guess the main problem would be with cross-compilation, but anyway I guess adhoc signatures only work on the same machine where they are generated.

This would be a major pain for projects distributing binaries. I wonder how Homebrew deals with it, since by default they install pre-built "Bottles".

What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

I don't know, but I can imagine 1) to normalize the execution path such that it always check signatures unconditionally, simplifying it and 2) to block binaries generated on other machines (and not properly signed by a trusted Developer ID) which before was sort of the role of the quarantine attributes.

@terinjokes
Copy link
Contributor

@terinjokes terinjokes commented Nov 18, 2020

You're right, I do get the binary. Interestingly, it comes out linker-signed but doesn't run, and codesign doesn't work on it.

My understanding is this is a bug already reported to Apple, see Homebrew/brew#9082 (comment)

I wonder how Homebrew deals with it, since by default they install pre-built "Bottles".

Per Homebrew/brew#7857 (comment) I don't think they're close to code signing bottles on ARM yet.

@rolandshoemaker
Copy link
Member

@rolandshoemaker rolandshoemaker commented Nov 18, 2020

  1. to block binaries generated on other machines (and not properly signed by a trusted Developer ID) which before was sort of the role of the quarantine attributes.

Looks like binaries aren't required to have a signature linked to a Developer ID if they were signed elsewhere. I was able to apply an ad-hoc signature to a binary on my Intel mac and transfer it to my M1 mac and it ran just fine. Appears like binaries just need a signature, doesn't really seem to matter where it came from.

@fxcoudert
Copy link

@fxcoudert fxcoudert commented Nov 18, 2020

Homebrew maintainer here:

the codesign_allocate helper tool cannot be found or used

That's indeed a bug in codesign, and Apple is aware

My understanding is this is a bug already reported to Apple, see Homebrew/brew#9082 (comment)

Yes. If this occurs, you need to change that file's inode (so copying it someplace and back will work). Once you've done that, you can sign and it will work. See Homebrew/brew#9102 for details.

I don't think they're close to code signing bottles on ARM yet

We do not distribute bottles yet, because our CI is not yet fully operational, but our codebase is otherwise ready. We apply ad-hoc signatures as part of formula installation: Homebrew/brew#9102

@fxcoudert
Copy link

@fxcoudert fxcoudert commented Nov 18, 2020

As for go, if Go uses its own linker (or anything other than Apple's ld) then it definitely needs to call codesign -s - on the binaries it produces: executables, shared libraries, etc.

@mislav
Copy link

@mislav mislav commented Nov 18, 2020

@FiloSottile (and others): thank you for the information! 🙇

Do you know if it is possible to use darwin/amd64 architecture to cross-compile binaries that are able to execute on darwin/arm64, providing that the binaries are explicitly signed?

I am interested in what are the options to distribute binaries for Go projects that work on Apple Silicon without having to actually compile them on Apple Silicon.

@bwesterb
Copy link

@bwesterb bwesterb commented Nov 18, 2020

@mislav Cross-compiling for darwin/arm64 works fine from darwin/amd64 and even linux/amd64. Obviously the resulting binaries are not signed. They will run if signed ad hoc on the same machine they will run. Binaries ad hoc signed on one machine didn't work on another machine for me as well.

@jameshartig
Copy link
Contributor

@jameshartig jameshartig commented Nov 18, 2020

Binaries ad hoc signed on one machine didn't work on another machine for me.

This comment seems to disagree: #42684 (comment)

I was able to apply an ad-hoc signature to a binary on my Intel mac and transfer it to my M1 mac and it ran just fine.

@bwesterb
Copy link

@bwesterb bwesterb commented Nov 18, 2020

Ok, I tested it again and now cross-signing does work. I might've transferred the wrong file earlier. Whoops.

@FiloSottile
Copy link
Member Author

@FiloSottile FiloSottile commented Nov 18, 2020

Codesigning from cmd/link should be possible. https://github.com/isignpy/isign by @neilk does that for iOS.

@networkimprov
Copy link

@networkimprov networkimprov commented Nov 18, 2020

The Zig project is also working on this here: ziglang/zig#7103 (thanks @komuw).

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 18, 2020

Thanks for the pointers. I'll look into them.

@fxcoudert
Copy link

@fxcoudert fxcoudert commented Nov 18, 2020

Binaries ad hoc signed on one machine didn't work on another machine for me

I can confirm that a linker or other ad hoc signature is sufficient to run the binary (or shared library) on any machine.

@aclements aclements added this to the Go1.16 milestone Nov 18, 2020
@randall77
Copy link
Contributor

@randall77 randall77 commented Nov 21, 2020

Not to throw a wrench in here, but anyone know what the interaction is with code signing + fat binaries?
Do you sign the individual binaries, or do you also/instead need to sign the container?
(I don't think there's any place in the format to sign the container, so I suspect the former.)

@markmentovai
Copy link

@markmentovai markmentovai commented Nov 21, 2020

Not to throw a wrench in here, but anyone know what the interaction is with code signing + fat binaries?

Each individual architecture “slice” has its own signature covering only that slice.

You can merge different thin Mach-O objects, already validly signed, into a single fat (universal) file, and it remains validly signed. You can also extract slices from a validly signed fat file, producing thin Mach-O objects that are validly signed. lipo is Apple’s tool to manipulate fat files. It’s a very simple archive format and is trivial to work with without lipo, too.

codesign --sign will also (re-)sign all architectures in a fat file.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 21, 2020

If it’s maintained by a separate tool, then a possible solution would be for that tool to also recompute and write a new ad-hoc linker-signed signature

My plan is #42684 (comment) . Basically what you said, generate the signature after stamping the buildid. Maybe we could do it both before and after stamping the buildid, so plain go tool link works.

I'm currently doing it as a separate tool just to make sure I understand the signing algorithm.

It is not a requirement to add this to an identifier.

Does it affect kernel's caching in some magical way? Or, what actually matters? The inode? When the kernel caches the signature's validity, at load time, or when then file is written (closed)? Does it require the binary to have a valid signature at every point of time? Thanks.

(LC_UUID can be rather valuable, so perhaps you’d consider adding it, although I agree that’s a distinct concern.)

Could you provide some information about LC_UUID, how it is useful and how it is generated? I'm happy to add it to the Go linker as long as I understand it. Thanks.

if you link the Go program with Apple's linker, it still won't run: GOOS=darwin GOARCH=arm64 go build -o test -ldflags '-linkmode=external -extldflags "-arch arm64"' ./program.go

This wouldn't be expected to work, because the Go toolchain modifies external-linker emitted binary in a few ways, stamping the buildid is one, and there are others.

@fayep
Copy link

@fayep fayep commented Nov 21, 2020

Buildid seems to be written into the binary by cmd/buildid. Perhaps on Darwin (This is all a platform feature which is disabled in intel Macs, but will be handled just fine by them, right?) it could invoke the signer passing the UUID maybe the LC_UUID if that's what it is for? And return the same on reading.

@jpap
Copy link
Contributor

@jpap jpap commented Nov 21, 2020

tl;dr: The basic ver 0x20001 CD is fine, with no UUID, no CD identifier hash suffix, so long as you copy the file after it is signed.

Thank you @markmentovai for the super-useful and detailed information. I can confirm that the UUID and CD identifier hash suffix doesn't affect the signature validation. I've updated my gist to include the UUID generation, in case you want to play with it.

There's something fishy about the way the final executable is run...

# No UUID, no CD identifier hash suffix
$ go run ./generate.go -work
Using gc: /Users/jpap/Development/go/src/github.com/golang/go/bin/go
Executable: test-639771800
Building Go program...
Codesign with codesign.go...
Run...
hello 639771800
$ ./test-639771800
zsh: killed     ./test-639771800
$ cp test-639771800 test-xxx
$ ./test-xxx 
hello 639771800
$ 

# With UUID, no CD identifier hash suffix
$ go run ./generate.go -work -cstool codesign.uuid.go
Using gc: /Users/jpap/Development/go/src/github.com/golang/go/bin/go
Executable: test-436445343
Building Go program...
Codesign with codesign.uuid.go...
Run...
hello 436445343
$ ./test-436445343
zsh: killed     ./test-436445343
$ cp test-436445343 test-yyy
$ ./test-yyy
hello 436445343
$

$ arch 
arm64
$

I don't understand why the binary runs within the generate.go program, but not outside it, unless you copy the file. Is it because generate.go is running under Rosetta? It gets even more weird with no signature at all, where the unsigned binary runs when launched form the Rosetta process, but otherwise is killed.

$ go run ./generate.go -work -cstool none
Using gc: /Users/jpap/Development/go/src/github.com/golang/go/bin/go
Executable: test-120056221
Building Go program...
Skipping codesign...
Run...
hello 120056221
$ ./test-120056221
zsh: killed     ./test-120056221
$ codesign -d -vvv test-120056221
test-120056221: code object is not signed at all
$ cp test-120056221 test-aaa
$ ./test-aaa
zsh: killed     ./test-aaa
$ file test-120056221
test-120056221: Mach-O 64-bit executable arm64
$

If you don't run the binary under Rosetta, you still need to make the copy. So it looks like @dmitshur is right about that... would love to know why!

$ go run ./generate.go -work -run=false
Using gc: /Users/jpap/Development/go/src/github.com/golang/go/bin/go
Executable: test-118293903
Building Go program...
Codesign with codesign.go...
$ ./test-118293903
zsh: killed     ./test-118293903
$ cp test-118293903 test-bbb
$ ./test-bbb
hello 118293903
$
@markmentovai
Copy link

@markmentovai markmentovai commented Nov 22, 2020

@cherrymui #42684 (comment):

  • Stamp the buildid. The buildid would be made to be insensitive to the LC_CODE_SIGNATURE blob. (but can be sensitive to the load command itself, which should not change)

I suggest making it sensitive to the LC_CODE_SIGNATURE load command, but skipping or blanking the dataoff and datasize fields and, of course as you say, the data that they point to. Rationale: try to keep the golang build ID insensitive to the Mach-O file being re-signed. While a simple replacement ad-hoc signature might not alter dataoff or datasize, a more complex re-signing could (if the identifier, requirements, entitlements, etc., change, or even if a different-version signature is used, or if the replacement signature is not ad-hoc and carries a CMS signature.) That said, this may wind up being hopeless anyway if the Mach-O file needs to grow to accommodate the new signature.

@cherrymui #42684 (comment):

Basically what you said, generate the signature after stamping the buildid. Maybe we could do it both before and after stamping the buildid, so plain go tool link works.

Yes, I’d favor striving to always have a legitimate linker-signed signature in place following any tool execution.

It is not a requirement to add this to an identifier.

Does it affect kernel's caching in some magical way? Or, what actually matters? The inode? When the kernel caches the signature's validity, at load time, or when then file is written (closed)? Does it require the binary to have a valid signature at every point of time? Thanks.

The identifier doesn’t enter xnu’s view of trust at all. There are higher-level user-space operations that do consider the identifier, but even then, it’s just an opaque string. For linker-signed purposes, I suggest setting the identifier to the filename of the Mach-O output file (just the basename, not the whole path), without any extra digest or other appendage.

In xnu, code signing is implemented within the VM system, as part of the unified buffer cache, and is attached at the vnode level. Pages are validated at fault time as they are brought in by the pager. 10.15.6 xnu-6153.141.1/osfmk/vm/vm_fault.c vm_fault_enter is the top of the funnel for most of this. You can trace that through to vm_page_validate_cs_mapped and the actual validation in bsd/kern/ubc_subr.c cs_validate_range.

All of this is to say that if you’re interacting with a file via mmap, you risk getting sniffed out. And golang’s linker does write its output to an mmaped region. There you go!

Is this a horrible bug? Absolutely, I think so! But I also think that it’s not an easy one to fix, because it’s based on a central design flaw. In cases where I’ve found legitimately terrible behaviors, like strip creating broken files that codesign wouldn’t be able to fix, Apple has responded rapidly with tool fixes, but the underlying architecture has mostly remained the same. I don’t love that it’s difficult or impossible to mmap something, even to handle as data, just because the kernel happened to sniff out something that it thought was a signature. Bugs abound in this area, I’m afraid.

So what can you do?

  • You can wait for your pages to leave the UBC, as @dmitshur found at #42684 (comment). If you don’t want to wait, purge can help. While that’s great for theory and testing, it’s not great for practice or production…or not requiring something to run as root. (purge is faster than a reboot, though, even on one of these new machines.)
  • As you’ve discovered, you can put your file contents into a new vnode (via a new inode), as long as you write the file with something like write and not writing to an mmaped region. cp or equivalent is fine for this, but the whole thing feels kind of brutish and I don’t love this solution.
  • You can avoid writing your file by writing to an mmaped region in the first place. Instead, issue writes to a file descriptor, or similar.
  • One of my favorites: call msync(…, MS_INVALIDATE) on the mmaped region, asking xnu to throw away what it knows about the vnode. You can even use this to “save” a broken vnode from an entirely different process by opening the file,mmaping it, and then calling msync.

ld64 manages to produce output that runs without getting killed (golang post-link modifications like the build ID notwithstanding), so what’s it doing? The most current source available is Xcode 11.3.1 ld64-530/src/ld/OutputFile.cpp ld::tool:: OutputFile::writeOutputFile, which is just a straight mmap in most common cases. But that’s old, and the new truth is that in Xcode 12.2 ld64-609.7, only on arm64, they’ve actually switched to a write approach in every case.

Could you provide some information about LC_UUID, how it is useful and how it is generated? I'm happy to add it to the Go linker as long as I understand it. Thanks.

This is going off in an unrelated direction, because LC_UUID isn’t really relevant to any of this, other than that it factors in to how codesign --sign creates a default identifier. Briefly: it’s the Mach-O equivalent of ELF’s .note.gnu.build-id, a way to uniquely identify a Mach-O file, which is typically used to correlate it with its out-of-band debug info (.dSYM). In Crashpad (that’s one of my things), I use LC_UUID to tie modules loaded on the client back to the original files and their debug info on the server. I think it would be good for the golang linker to write this load command, but it won’t help here at all, and it’s a totally different issue.

@jpap #42684 (comment):

tl;dr: The basic ver 0x20001 CD is fine, with no UUID, no CD identifier hash suffix, so long as you copy the file after it is signed.

Sure, but I suggest sticking with 0x20400 for a linker-signed signature. This is what Xcode 12.2 ld64 does, and that should be treated as the baseline linker for mac-arm64. Apple would be within their rights to require this as a minimum on mac-arm64, even if they aren’t doing so now, and even though it would be rude of them to break golang output in existence at the time they’d make such a change, I can’t say that this would be something that they’d catch early in testing, and I know that this is something that’s happened before.

What about the zeroes vs. real execSegBase, execSegLimit, and execSegFlags fields? 10.15 codesign --sign leaves these at zero when targeting macOS, but sets them properly for all of Apple’s other operating systems (iOS, etc.) 10.15.6 Security-59306.140.5/OSX/libsecurity_codesigning/lib/machorep.cpp Security::CodeSigning::MachORep::needsExecSeg This has changed in 11.0 (and in Xcode 12.2 ld64 writing linker-signed signatures), which always set them to valid values, even for macOS. While it’s possible and supported to codesign --sign targeting mac-arm64 from a 10.15 host, linker-signed output for mac-arm64 will never omit valid values for these fields, and it’s incredibly easy to set them properly.

Definitely no LC_UUID is required, and the identifier ought not be considered for any practical purpose, particularly for ad-hoc signatures.

There's something fishy about the way the final executable is run...

Yes, invoking Rosetta is basically an end run around the new code signature enforcement that Apple is trying hard to impose for arm64. In order to maintain maximum x86_64 compatibility, some of the new things that come with arm64 aren’t available in the same way for x86_64. One of these things is the requirement that all code be signed. Since xnu is responsible for enforcement, it’s natural that some of these compatibility concessions were made in xnu. In this case, the signature check at execution time is fired off from within 10.15.6 xnu-6153.141.1/bsd/kern/mach_loader.c parse_machfile and load_code_signature. This is 10.15.6 source, and I haven’t seen 11.0 xnu source yet to confirm exactly how this is hooked up for arm64, but it’s perfectly reasonable that when the parent is running under Rosetta, the initial signature check of the new child executable would be conducted under Rosetta rules, regardless of whether the new executable is arm64 or x86_64. That’d make this a bug, and it’s worth a feedback report or, even better, a product-security contact.

mark@arm-and-hammer zsh% clang -arch arm64 -x c - -o /tmp/no_signature -Wl,-no_adhoc_codesign <<< 'int main(int argc, char* argv[]) { return 0; }'
mark@arm-and-hammer zsh% /tmp/no_signature; echo $?               
zsh: killed     /tmp/no_signature
137
mark@arm-and-hammer zsh% arch -arch x86_64 sh -c /tmp/no_signature; echo $?  
0

Bonus information: this form of code signature enforcement isn’t strictly new, it’s been available on x86_64 for a long time as kill semantics (an option to codesign --sign --options) and is also enabled under the hardened runtime (option runtime). But because you won’t find linker output bearing a signature with that option, you won’t really experience the bug on x86_64. If you’re interested, you can mimic the bug on x86_64, even on macOS 10.15, by signing an executable with codesign --sign=- --options=kill and then having something copy it by writing it to a file-backed mmaped region. You’ll find that you can’t run the copy, the same behavior that we now see on arm64 by default.

If you don't run the binary under Rosetta, you still need to make the copy. So it looks like @dmitshur is right about that... would love to know why!

All answered above!

And I know it’s just for testing, but having been down this road recently: don’t rely on the presence of Rosetta, it’s not part of the base OS and may not be there at all!

@jpap
Copy link
Contributor

@jpap jpap commented Nov 22, 2020

Phenomenal -- thank you for all of the detail @markmentovai, I think I will sleep better tonight. ❤️

I suggest making it sensitive to the LC_CODE_SIGNATURE load command, but skipping or blanking the dataoff and datasize fields and, of course as you say, the data that they point to. Rationale: try to keep the golang build ID insensitive to the Mach-O file being re-signed. While a simple replacement ad-hoc signature might not alter dataoff or datasize, a more complex re-signing could (if the identifier, requirements, entitlements, etc., change, or even if a different-version signature is used, or if the replacement signature is not ad-hoc and carries a CMS signature.) That said, this may wind up being hopeless anyway if the Mach-O file needs to grow to accommodate the new signature.

The executable will definitely grow when using a CMS encoded signature (which usually also comes with a set of requirements, and on iOS, entitlements), unless you include enough padding up front. Having a go linker option to request padding could be useful if it means a go build can go faster on a cache hit. (This would be analogous to the codesign_allocate step that's part of an Xcode build, where you can opt for an over-allocation.) Eating an extra link on cache miss isn't too bad, but gets worse with the external linker which is a lot slower. Either way, it's not a dealbreaker.

Basically what you said, generate the signature after stamping the buildid. Maybe we could do it both before and after stamping the buildid, so plain go tool link works.

Yes, I’d favor striving to always have a legitimate linker-signed signature in place following any tool execution.

I'm also in favor of this, so long as there is an opt-out flag. It saves having a custom toolchain that re-signs the binary with a CMS encoded signature from removing the existing signature first: not a big deal, but it would save me some work. ;) Apple's linker supports this with the -no_adhoc_codesign flag; it would be nice for the go linker to have a similar option.

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 22, 2020

Change https://golang.org/cl/272257 mentions this issue: cmd/internal/buildid: update Mach-O code signature when rewriting buildid

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 22, 2020

Change https://golang.org/cl/272255 mentions this issue: cmd/internal/buildid: exclude Mach-O code signature in hash calculation

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 22, 2020

Change https://golang.org/cl/272256 mentions this issue: cmd/link: code-sign on darwin/arm64

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 22, 2020

Change https://golang.org/cl/272254 mentions this issue: cmd/internal/codesign: new package

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 22, 2020

The CLs above let the toolchain emit signed binaries. Could someone give it a try and see if it works? If not, does it work after copying the binary? (I wrote them yesterday without seeing @markmentovai 's comment. I'll play with the msync. Thanks :)

The CLs can be checked out by doing

git fetch https://go.googlesource.com/go refs/changes/57/272257/1 && git checkout FETCH_HEAD

and rebuild the toolchain.

Thanks!

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 22, 2020

@markmentovai thanks very much for the detailed reply!

Yes, I’d favor striving to always have a legitimate linker-signed signature in place following any tool execution.

Yes, this is what it does with the current version of the CLs.

Sure, but I suggest sticking with 0x20400 for a linker-signed signature.

and this as well :)

skipping or blanking the dataoff and datasize fields

Yeah, sounds like a good idea. If I understand it correctly, it probably rarely matters, though. Unless one explicitly does go tool buildid -w after a non-ad-hoc signing, or non-ad-hoc signs a toolchain binary (compiler/linker/assemlber/etc.).

LC_UUID isn’t really relevant to any of this

Yeah. Just asked while you are here :) Thanks.

@fayep
Copy link

@fayep fayep commented Nov 22, 2020

git fetch https://go.googlesource.com/go refs/changes/57/272257/1 && git checkout FETCH_HEAD

I bootstrapped with this source successfully from x86_64 to arm64 creating signed binaries. I then attempted to build all.bash with that bootstrap Go and it failed.

The apps behaved as though they were unsigned. I checked the signatures and they were reported good, so based on the thread and experience, I copied the toolchain tree to another directory (could also have extracted the tarball) and it then compiled dist. However dist suffered the same issue of stale inode signature cache. It did appear successfully signed and copying dist to another location made it work.

Congrats. We just need to move then copy the file back as part of the process and we’re done.

@FiloSottile
Copy link
Member Author

@FiloSottile FiloSottile commented Nov 22, 2020

The binaries work but only after copying them.

➜  ~ GOARCH=arm64 gotip download 272257
This will download and execute code from golang.org/cl/272257, continue? [y/n] y
Fetching CL 272257, Patch Set 1...
From https://go.googlesource.com/go
 * branch            refs/changes/57/272257/1 -> FETCH_HEAD
HEAD is now at 376d052 cmd/internal/buildid: update Mach-O code signature when rewriting buildid
Building Go cmd/dist using /usr/local/Cellar/go/1.15.5/libexec. (go1.15.5 darwin/amd64)
Building Go toolchain1 using /usr/local/Cellar/go/1.15.5/libexec.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, darwin/amd64.
Building packages and commands for target, darwin/arm64.
---
Installed Go for darwin/arm64 in /Users/filippo/sdk/gotip
Installed commands in /Users/filippo/sdk/gotip/bin
Success. You may now run 'gotip'!

➜  ~ /Users/filippo/sdk/gotip/bin/darwin_arm64/go version
[1]    3317 killed     /Users/filippo/sdk/gotip/bin/darwin_arm64/go version

➜  ~ codesign -vv /Users/filippo/sdk/gotip/bin/darwin_arm64/go
/Users/filippo/sdk/gotip/bin/darwin_arm64/go: valid on disk
/Users/filippo/sdk/gotip/bin/darwin_arm64/go: satisfies its Designated Requirement

➜  ~ codesign -dv /Users/filippo/sdk/gotip/bin/darwin_arm64/go
Executable=/Users/filippo/sdk/gotip/bin/darwin_arm64/go
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=104606 flags=0x20002(adhoc,linker-signed) hashes=3266+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

➜  ~ cp /Users/filippo/sdk/gotip/bin/darwin_arm64/go ~/tmp/go

➜  ~ ~/tmp/go version
go version devel +376d052 Sun Nov 22 01:08:02 2020 -0500 darwin/arm64
@oxisto
Copy link

@oxisto oxisto commented Nov 22, 2020

The CLs above let the toolchain emit signed binaries. Could someone give it a try and see if it works? If not, does it work after copying the binary? (I wrote them yesterday without seeing @markmentovai 's comment. I'll play with the msync. Thanks :)

The CLs can be checked out by doing

git fetch https://go.googlesource.com/go refs/changes/57/272257/1 && git checkout FETCH_HEAD

and rebuild the toolchain.

Thanks!

They are signed correctly, but only work after copying.

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 22, 2020

Change https://golang.org/cl/272258 mentions this issue: cmd/link: invalidate kernel cache on darwin

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 22, 2020

Thanks all for confirming!

Does CL https://go-review.googlesource.com/c/go/+/272258 help? Thanks!

To check out,

git fetch https://go.googlesource.com/go refs/changes/58/272258/1 && git checkout FETCH_HEAD
@FiloSottile
Copy link
Member Author

@FiloSottile FiloSottile commented Nov 22, 2020

Winner winner, chicken dinner! Great job @cherrymui and thank you to @markmentovai and everyone who shared insights!

$ GOARCH=arm64 gotip download 272258
$ ~/sdk/gotip/bin/darwin_arm64/go version
go version devel +7716a2f Sun Nov 22 11:44:49 2020 -0500 darwin/arm64
$ ~/sdk/gotip/bin/darwin_arm64/go test
PASS
ok  	filippo.io/edwards25519	9.822s
@oxisto
Copy link

@oxisto oxisto commented Nov 22, 2020

Thanks all for confirming!

Does CL https://go-review.googlesource.com/c/go/+/272258 help? Thanks!

To check out,

git fetch https://go.googlesource.com/go refs/changes/58/272258/1 && git checkout FETCH_HEAD

Can also confirm that it works now without copying. Brilliant. Excellent work, you just made me an even happier M1 owner :)

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Nov 22, 2020

Thanks all for trying out and sharing feedbacks! And thank you @markmentovai for the detailed information!

@bwesterb
Copy link

@bwesterb bwesterb commented Nov 22, 2020

Can confirm it works and passes Go's tests. Thanks!

For those who want all the commands to try themselves — run the following on another machine with Go installed:

git clone https://go.googlesource.com/go
cd go
git fetch https://go.googlesource.com/go refs/changes/58/272258/1 && git checkout FETCH_HEAD
cd src
GOOS=darwin GOARCH=arm64 ./bootstrap.bash

This will create a go-darwin-arm64-bootstrap.tbz with a Go distribution that should work on M1.

@fayep
Copy link

@fayep fayep commented Nov 22, 2020

You can do that all on the m1 mac by replacing the last line with
arch --x86_64 env GODEBUG=asyncpreemptoff=1 GOOS=darwin GOARCH=arm64 ./bootstrap.bash

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Nov 22, 2020

Does CL https://go-review.googlesource.com/c/go/+/272258 help? Thanks!

As others confirmed above, that CL indeed makes it possible to do go run hello.go and have it work as usual.

That CL enabled me to try running all.bash, which has passed at least once:

src $ ./all.bash
Building Go cmd/dist using /Users/dmitri/go116r2. (devel +7716a2fbb7 Sun Nov 22 11:44:49 2020 -0500 darwin/arm64)
[...]
ALL TESTS PASSED
---
Installed Go for darwin/arm64 in /Users/dmitri/gotip
Installed commands in /Users/dmitri/gotip/bin
*** You need to add /Users/dmitri/gotip/bin to your PATH.

I'm seeing frequent runs that fail due to a sporadic "signal: illegal instruction". Filed #42774 that collects some of those failures, if it helps.

@matloob
Copy link
Contributor

@matloob matloob commented Nov 23, 2020

I can also verify that CL 272258 works. On a fresh M1 MacBook Air I installed XCode (for some reason just installing command line tools wasn't enough and used the 1.15 Go installer to get a bootstrap Go install (through Rosetta 2).
Then GODEBUG=asyncpreemptoff=1 GOARCH=arm64 ./make.bash then I replaced $GOROOT/bin/go and $GOROOT/bin/gofmt with the binaries in the $GOROOT/bin/darwin_arm64 directory. (I then re-bootstrapped off of a copy of the built darwin/arm64 GOROOT for good measure).

Things work pretty well!

@markmentovai
Copy link

@markmentovai markmentovai commented Nov 23, 2020

Following up:

I reported the bug encountered when writing to the executable via mmap (without msync(…, MS_INVALIDATE)) to Apple as FB8914231, with an OpenRadar copy.

Possibly of interest, an unrelated effort this weekend also resulted in the filing of FB8914243, a similar bug about signatures remaining attached to vnodes even when the file contents change.

These are longstanding and difficult-to-fix bugs, but arm64’s requirement that all code must be signed has raised visibility, and its requirement to impose “kill” semantics globally has raised severity, so I re-filed with new information.

I also reported the technique to circumvent the requirement that all arm64 code be signed as a security bug, currently embargoed but eventually a copy will be visible here.

@fxcoudert
Copy link

@fxcoudert fxcoudert commented Nov 23, 2020

@markmentovai I am wondering whether your FB8914231 is not a different manifestation of the same bug we have seen and reported in the past, in some cases, with Apple's own tools: #42684 (comment)

@markmentovai
Copy link

@markmentovai markmentovai commented Nov 24, 2020

@fxcoudert: FB8914231 is part of what’s happening in Homebrew/brew#9082 (comment). When you invoke the compiler under Rosetta (arch -arch x86_64), its children, including the linker, will also run under Rosetta if possible.

In #42684 (comment), I said:

ld64 manages to produce output that runs without getting killed (golang post-link modifications like the build ID notwithstanding), so what’s it doing? The most current source available is Xcode 11.3.1 ld64-530/src/ld/OutputFile.cpp ld::tool::OutputFile::writeOutputFile, which is just a straight mmap in most common cases. But that’s old, and the new truth is that in Xcode 12.2 ld64-609.7, only on arm64, they’ve actually switched to a write approach in every case.

The operative part that I left in italics is significant here. When you run the x86_64 slice of ld64 under Rosetta, it still writes output to an mmaped region when possible, instead of using write always. The UBC doesn’t know or care whether the writes came from arm64 or translated x86_64 code, so the bug manifests as in this bug, #42684, and FB8914231.

You may also be seeing signs of FB8914243 in there, particularly if, during testing, you’re replacing the contents of Mach-O files by recycling the inode rather than forcing a new one. (Not that there should be anything wrong with doing so.)

More generally, once you have a Mach-O image in a vnode that xnu has decided is “bad”, it’ll suffer from the immediate SIGKILL that we’ve been discussing, and attempts to re-sign it in place with codesign --sign will result in the confusing message the codesign_allocate helper tool cannot be found or used, which is not entirely accurate. There was an earlier bug in this area that we encountered during the beta period, so we didn’t speak publicly of it, but I think it’s fine to share now that the beta period is over, so I brought it into OpenRadar: FB8735191. It’s relevant because it discusses how codesign_allocate works and why codesign --sign reports that confusing message when you’re suffering from FB8914231, FB8914243, or other related bugs. In the original FB8735191 report, innocuous use of strip and install_name_tool was another vector to produce unmappable vnodes, and it was [incompletely] resolved by making those tools replace linker-signed code signatures with new ones, but notably the questionable underlying behavior of vm_fault_enter was left intact, giving rise to this new series of bugs.

If you have more questions not specific to golang, feel free to @ me in another bug report, and I’ll see what else I can answer. I don’t want to fill up golang’s bug tracker with too much unrelated noise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.