all: ensure that Apple’s notarization requirements are met

[artyom]

What version of Go are you using (go version)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/artyom/Library/Caches/go-build"
GOENV="/Users/artyom/Library/Application Support/go/env"
GOEXE=""
GOFLAGS="-ldflags=-w -trimpath"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/artyom/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/artyom/Library/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/artyom/Library/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/artyom/Repositories/artyom/rex/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lb/3rk8rqs53czgb4v35w_342xc0000gn/T/go-build499526677=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Installed go from https://dl.google.com/go/go1.13.3.darwin-amd64.tar.gz and then tried to use it to build project. This is on macOS Catalina 10.15:

Darwin MacBook-Air.local 19.0.0 Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64 x86_64

What did you expect to see?

Go works as usual.

What did you see instead?

On first go call I'm presented with OS pop-up that "macOS cannot verify that this app is free from malware", with options either to discard this window (the process is killed then) or move the binary to trash. After whitelisting this binary via Preferences → Security & Privacy → General → Allow Anyway, go can be called fine.

Next I tried to build project with go install and got the came pop-up for "compile" tool.

[artyom]

To be able to run updated Go 1.13.3 and compile my project with it, I had to whitelist so far these binaries:

• go
• compile
• asm
• cgo
• link

Whitelisting each takes the following:

1. presented with os pop-up about binary not being verified;
2. click cancel, at this moment binary is killed;
3. go to Preferences → Security & Privacy → General, click "Allow Anyway" button;
4. try running the same command again;
5. presented with os pop-up about binary not being verified, but now with another option to run binary anyway;
6. click "open" button on that pop-up.

[bradfitz]

Good times.

/cc @golang/osp-team

[bradfitz]

Related: #34748 (we need a Catalina builder)

[networkimprov]

Discussion on Hacker News: https://news.ycombinator.com/item?id=21179970

Also, Catalina is reported to be a disastrous upgrade.

[andybons]

I’ll take a look at this.

[dmitshur]

I can reproduce on my personal MBP with macOS 10.15 (19A602) when installing from the .pkg installer too, however it's a different (less intrusive) experience.

The "macOS cannot verify that this app is free from malware" prompt appears when first running the installer. After accepting and installing, Go binaries such as go, compile, do not produce the same prompt. Also, re-running the .pkg installer doesn't ask again.

[andybons]

In the process of submitting the installer to Apple's notary service, but the process is very slow due to a service issue on Apple's side and our internal tooling keeps timing out. Diagnosing this has been difficult.

It may be that we will need to enable the Hardened Runtime which likely requires a newer version of Xcode that we're using to build releases and an audit of whatever entitlements are required for our binaries to run.

@aclements can you take a look at the link above and let me know which you think we'll need (if any)?

[aclements]

@andybons, I don't think we need any of the runtime exceptions from that page. The only one that we may even potentially want is "Disable Library Validation Entitlement", which would let users use the the plugin package without signing their plugins, but it seems like developers probably should sign their plugins.

[vp2177]

The (.pkg) installer doesn't even start for me. Is that the same issue, or unrelated?

"go1.13.3.darwin-amd64.pkg can't be opened because Apple cannot check it for malicious software."

Meanwhile, the older Go version I had installed from before the OS X Catalina update, continues to work fine, no warning dialogs.

[andybons]

@vp2177 it’s the same issue.

[andybons]

Good news: according to Apple, we don’t have to enable the hardened runtime until January 2020. That said, we’ll likely have to start signing the binaries themselves (not just the installer as we do now), so this issue should stay open even if we fix the problem in the short term via notarization of the installer.

[andybons]

@dmitshur to follow up on your experience, do you have SIP (System Integrity Protection) turned on? csrutil status should give you that info.

@artyom do you have SIP turned on? Thanks.

[dmitshur]

@andybons Yes.

$ csrutil status
System Integrity Protection status: enabled.

[artyom]

@andybons yes, it's enabled:

¶ csrutil status
System Integrity Protection status: enabled.

[andybons]

Our installer is getting stuck in limbo with the Notarization service. We’ve contacted Apple and are awaiting a response. It’s unclear what the ETA on that will be, though.

[andybons]

https://developer.apple.com/system-status/ now shows an issue with the Notary service:

Performance: Today, 10:10 AM - ongoing

Some users are affected

Users are experiencing a problem with this service. We are working to resolve this issue.

[andybons]

Notarization of the installer package has completed successfully. We are testing locally to ensure no scary popups are shown.

[andybons]

While we believe we have the installer package issue resolved, we have to deal with the .tar.gz bundles, which require notarization of each individual binary. Working on this.

[andybons]

As an update, this is what we are going to do:

• Sign all Mach-O binaries within $GOROOT/bin/ and $GOROOT/pkg/tool/darwin_amd64 (without the Hardened Runtime enabled as it’s not required just yet)
• Construct the .pkg installer with the signed binaries included
• Sign the .pkg installer
• Send the .pkg installer off to the notarization service and staple the resulting ticket to the installer

Since the notarization process is recursive (and the bits being distributed in the .tar.gz file are the same that will be in the installer), this should solve the warnings for users that choose to install using either the .pkg file or the .tar.gz file.

I will update this thread once this work is complete.

[andybons]

The initial work mentioned in #34986 (comment) is now complete. We will likely issue a point release this week with the changes after some further testing.

[virskor]

same issue

csrutil status
System Integrity Protection status: disabled.

go version
go version go1.13.3 darwin/amd64

osversion
10.15.1 (19B88)

[andybons]

New point releases have been released (https://golang.org/dl/#go1.13.4) that have signed and notarized binaries/installers. From testing locally I did not see any issues but feel free to comment if there are further problems.

Will continue the rest of the work (enabling the Hardened Runtime, for example) on #31918.

[andybons]

Oh, and Happy Halloween! 👻

[andybons]

Re-opening as #31918 should be tracking whether binaries created by the Go toolchain remain acceptable to Apple’s notarization service, which is a different issue than the toolchain itself being acceptable.

Remaining work:

• Enable the Hardened Runtime for the go, gofmt, link, etc. binaries (and figure out whatever entitlements are needed)
• Figure out what to do about binaries shipped (within testdata) that will never be signed but the notary will reject

[andybons]

Below is derived from the log returned by Apple’s notarization service. I expect any warning to turn into an error come January 2020:

Warnings we intend to fix by enabling the Hardened Runtime on the listed binaries:

[warning]: (go/bin/go): The executable does not have the hardened runtime enabled.
[warning]: (go/bin/gofmt): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/vet): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/objdump): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/asm): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/trace): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/pprof): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/dist): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/pack): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/addr2line): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/compile): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/link): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/cgo): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/nm): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/fix): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/test2json): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/doc): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/buildid): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/cover): The executable does not have the hardened runtime enabled.

Warnings we’ll have to fix another way:

[warning]: (go/src/compress/gzip/testdata/issue6550.gz): b'gunzip: data stream error\ngunzip: /tmp/tmpd_lw2hk0/issue6550.gz.unpacked_00/issue6550.gz: uncompress failed\n'
[warning]: (go/src/archive/zip/testdata/go-no-datadesc-sig.zip): b"ditto: foo.txt: No such file or directory\nditto: Couldn't read pkzip signature.\n"
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The binary is not signed.
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The signature does not include a secure timestamp.
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The executable does not have the hardened runtime enabled.

De-duping from the above, the following binaries need to be dealt with (through obfuscation or some other means):

go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath
go/src/archive/zip/testdata/go-no-datadesc-sig.zip
go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec
go/src/debug/macho/testdata/gcc-amd64-darwin-exec
go/src/compress/gzip/testdata/issue6550.gz
go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath
go/src/cmd/internal/buildid/testdata/a.macho
go/src/debug/macho/testdata/gcc-386-darwin-exec

[aclements]

Is it saying the binaries need any particular hardened runtime entitlements, or just that all binaries need to be signed as having a hardened runtime?

[andybons]

@aclements just that the binaries must have the hardened runtime enabled. It doesn’t (and likely won‘t/can’t?) say what entitlements a binary needs.