Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: ensure that Apple’s notarization requirements are met #34986

Open
artyom opened this issue Oct 18, 2019 · 27 comments
Assignees
Milestone

Comments

@artyom
Copy link
Member

@artyom artyom commented Oct 18, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/artyom/Library/Caches/go-build"
GOENV="/Users/artyom/Library/Application Support/go/env"
GOEXE=""
GOFLAGS="-ldflags=-w -trimpath"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/artyom/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/artyom/Library/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/artyom/Library/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/artyom/Repositories/artyom/rex/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lb/3rk8rqs53czgb4v35w_342xc0000gn/T/go-build499526677=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Installed go from https://dl.google.com/go/go1.13.3.darwin-amd64.tar.gz and then tried to use it to build project. This is on macOS Catalina 10.15:

Darwin MacBook-Air.local 19.0.0 Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64 x86_64

What did you expect to see?

Go works as usual.

What did you see instead?

On first go call I'm presented with OS pop-up that "macOS cannot verify that this app is free from malware", with options either to discard this window (the process is killed then) or move the binary to trash. After whitelisting this binary via Preferences → Security & Privacy → General → Allow Anyway, go can be called fine.

Next I tried to build project with go install and got the came pop-up for "compile" tool.

Screen Shot 2019-10-18 at 13 00 17

Screen Shot 2019-10-18 at 13 00 23

@artyom

This comment has been minimized.

Copy link
Member Author

@artyom artyom commented Oct 18, 2019

To be able to run updated Go 1.13.3 and compile my project with it, I had to whitelist so far these binaries:

  • go
  • compile
  • asm
  • cgo
  • link

Whitelisting each takes the following:

  1. presented with os pop-up about binary not being verified;
  2. click cancel, at this moment binary is killed;
  3. go to Preferences → Security & Privacy → General, click "Allow Anyway" button;
  4. try running the same command again;
  5. presented with os pop-up about binary not being verified, but now with another option to run binary anyway;
  6. click "open" button on that pop-up.
@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Oct 18, 2019

Good times.

/cc @golang/osp-team

@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Oct 18, 2019

Related: #34748 (we need a Catalina builder)

@networkimprov

This comment has been minimized.

Copy link

@networkimprov networkimprov commented Oct 18, 2019

Discussion on Hacker News: https://news.ycombinator.com/item?id=21179970

Also, Catalina is reported to be a disastrous upgrade.

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 18, 2019

I’ll take a look at this.

@andybons andybons self-assigned this Oct 18, 2019
@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Oct 18, 2019

I can reproduce on my personal MBP with macOS 10.15 (19A602) when installing from the .pkg installer too, however it's a different (less intrusive) experience.

The "macOS cannot verify that this app is free from malware" prompt appears when first running the installer. After accepting and installing, Go binaries such as go, compile, do not produce the same prompt. Also, re-running the .pkg installer doesn't ask again.

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 21, 2019

In the process of submitting the installer to Apple's notary service, but the process is very slow due to a service issue on Apple's side and our internal tooling keeps timing out. Diagnosing this has been difficult.

It may be that we will need to enable the Hardened Runtime which likely requires a newer version of Xcode that we're using to build releases and an audit of whatever entitlements are required for our binaries to run.

@aclements can you take a look at the link above and let me know which you think we'll need (if any)?

@aclements

This comment has been minimized.

Copy link
Member

@aclements aclements commented Oct 21, 2019

@andybons, I don't think we need any of the runtime exceptions from that page. The only one that we may even potentially want is "Disable Library Validation Entitlement", which would let users use the the plugin package without signing their plugins, but it seems like developers probably should sign their plugins.

@vp2177

This comment has been minimized.

Copy link

@vp2177 vp2177 commented Oct 22, 2019

The (.pkg) installer doesn't even start for me. Is that the same issue, or unrelated?

Screenshot 2019-10-18 at 17 10 09

"go1.13.3.darwin-amd64.pkg can't be opened because Apple cannot check it for malicious software."

Meanwhile, the older Go version I had installed from before the OS X Catalina update, continues to work fine, no warning dialogs.

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 22, 2019

@vp2177 it’s the same issue.

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 22, 2019

Good news: according to Apple, we don’t have to enable the hardened runtime until January 2020. That said, we’ll likely have to start signing the binaries themselves (not just the installer as we do now), so this issue should stay open even if we fix the problem in the short term via notarization of the installer.

@andybons andybons pinned this issue Oct 22, 2019
@andybons andybons added the OS-Darwin label Oct 22, 2019
@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 22, 2019

@dmitshur to follow up on your experience, do you have SIP (System Integrity Protection) turned on? csrutil status should give you that info.

@artyom do you have SIP turned on? Thanks.

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Oct 22, 2019

@andybons Yes.

$ csrutil status
System Integrity Protection status: enabled.
@artyom

This comment has been minimized.

Copy link
Member Author

@artyom artyom commented Oct 22, 2019

@andybons yes, it's enabled:

¶ csrutil status
System Integrity Protection status: enabled.
@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 22, 2019

Our installer is getting stuck in limbo with the Notarization service. We’ve contacted Apple and are awaiting a response. It’s unclear what the ETA on that will be, though.

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 22, 2019

https://developer.apple.com/system-status/ now shows an issue with the Notary service:

Performance: Today, 10:10 AM - ongoing

Some users are affected

Users are experiencing a problem with this service. We are working to resolve this issue.

@andybons andybons changed the title Go 1.13.3 distribution lacks notarization for macOS Catalina, requiring whitelisting of each binary Notarize our installer and binaries (required as of macOS 10.15 Catalina) Oct 22, 2019
@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 23, 2019

Notarization of the installer package has completed successfully. We are testing locally to ensure no scary popups are shown.

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 23, 2019

While we believe we have the installer package issue resolved, we have to deal with the .tar.gz bundles, which require notarization of each individual binary. Working on this.

@rsc rsc changed the title Notarize our installer and binaries (required as of macOS 10.15 Catalina) macOS 10.15 Catalina rejects non-notarized Go installer, binaries Oct 24, 2019
@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 24, 2019

As an update, this is what we are going to do:

  • Sign all Mach-O binaries within $GOROOT/bin/ and $GOROOT/pkg/tool/darwin_amd64 (without the Hardened Runtime enabled as it’s not required just yet)
  • Construct the .pkg installer with the signed binaries included
  • Sign the .pkg installer
  • Send the .pkg installer off to the notarization service and staple the resulting ticket to the installer

Since the notarization process is recursive (and the bits being distributed in the .tar.gz file are the same that will be in the installer), this should solve the warnings for users that choose to install using either the .pkg file or the .tar.gz file.

I will update this thread once this work is complete.

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Oct 29, 2019

The initial work mentioned in #34986 (comment) is now complete. We will likely issue a point release this week with the changes after some further testing.

@rsc rsc unpinned this issue Oct 30, 2019
@rsc rsc pinned this issue Oct 30, 2019
@virskor

This comment has been minimized.

Copy link

@virskor virskor commented Oct 31, 2019

same issue

csrutil status
System Integrity Protection status: disabled.

go version
go version go1.13.3 darwin/amd64

osversion
10.15.1 (19B88)
@ianthehat ianthehat unpinned this issue Oct 31, 2019
@andybons andybons pinned this issue Oct 31, 2019
@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Nov 1, 2019

New point releases have been released (https://golang.org/dl/#go1.13.4) that have signed and notarized binaries/installers. From testing locally I did not see any issues but feel free to comment if there are further problems.

Will continue the rest of the work (enabling the Hardened Runtime, for example) on #31918.

@andybons andybons closed this Nov 1, 2019
@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Nov 1, 2019

Oh, and Happy Halloween! 👻

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Nov 1, 2019

Re-opening as #31918 should be tracking whether binaries created by the Go toolchain remain acceptable to Apple’s notarization service, which is a different issue than the toolchain itself being acceptable.

Remaining work:

  • Enable the Hardened Runtime for the go, gofmt, link, etc. binaries (and figure out whatever entitlements are needed)
  • Figure out what to do about binaries shipped (within testdata) that will never be signed but the notary will reject
@andybons andybons reopened this Nov 1, 2019
@andybons andybons changed the title macOS 10.15 Catalina rejects non-notarized Go installer, binaries Ensure that Apple’s notarization requirements are met Nov 1, 2019
@andybons andybons added NeedsFix and removed NeedsInvestigation labels Nov 1, 2019
@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Nov 1, 2019

Below is derived from the log returned by Apple’s notarization service. I expect any warning to turn into an error come January 2020:

Warnings we intend to fix by enabling the Hardened Runtime on the listed binaries:

[warning]: (go/bin/go): The executable does not have the hardened runtime enabled.
[warning]: (go/bin/gofmt): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/vet): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/objdump): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/asm): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/trace): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/pprof): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/dist): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/pack): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/addr2line): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/compile): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/link): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/cgo): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/nm): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/fix): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/test2json): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/doc): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/buildid): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/cover): The executable does not have the hardened runtime enabled.

Warnings we’ll have to fix another way:

[warning]: (go/src/compress/gzip/testdata/issue6550.gz): b'gunzip: data stream error\ngunzip: /tmp/tmpd_lw2hk0/issue6550.gz.unpacked_00/issue6550.gz: uncompress failed\n'
[warning]: (go/src/archive/zip/testdata/go-no-datadesc-sig.zip): b"ditto: foo.txt: No such file or directory\nditto: Couldn't read pkzip signature.\n"
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The binary is not signed.
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The signature does not include a secure timestamp.
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The executable does not have the hardened runtime enabled.

De-duping from the above, the following binaries need to be dealt with (through obfuscation or some other means):

go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath
go/src/archive/zip/testdata/go-no-datadesc-sig.zip
go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec
go/src/debug/macho/testdata/gcc-amd64-darwin-exec
go/src/compress/gzip/testdata/issue6550.gz
go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath
go/src/cmd/internal/buildid/testdata/a.macho
go/src/debug/macho/testdata/gcc-386-darwin-exec
@aclements

This comment has been minimized.

Copy link
Member

@aclements aclements commented Nov 2, 2019

Is it saying the binaries need any particular hardened runtime entitlements, or just that all binaries need to be signed as having a hardened runtime?

@andybons

This comment has been minimized.

Copy link
Member

@andybons andybons commented Nov 3, 2019

@aclements just that the binaries must have the hardened runtime enabled. It doesn’t (and likely won‘t/can’t?) say what entitlements a binary needs.

@odeke-em odeke-em changed the title Ensure that Apple’s notarization requirements are met all: ensure that Apple’s notarization requirements are met Nov 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.