Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

all: ensure that Go toolchain meets Apple’s notarization requirements #34986

Closed
artyom opened this issue Oct 18, 2019 · 43 comments
Closed

all: ensure that Go toolchain meets Apple’s notarization requirements #34986

artyom opened this issue Oct 18, 2019 · 43 comments
Assignees
Milestone

Comments

@artyom
Copy link
Member

@artyom artyom commented Oct 18, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/artyom/Library/Caches/go-build"
GOENV="/Users/artyom/Library/Application Support/go/env"
GOEXE=""
GOFLAGS="-ldflags=-w -trimpath"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/artyom/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/artyom/Library/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/artyom/Library/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/artyom/Repositories/artyom/rex/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lb/3rk8rqs53czgb4v35w_342xc0000gn/T/go-build499526677=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Installed go from https://dl.google.com/go/go1.13.3.darwin-amd64.tar.gz and then tried to use it to build project. This is on macOS Catalina 10.15:

Darwin MacBook-Air.local 19.0.0 Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64 x86_64

What did you expect to see?

Go works as usual.

What did you see instead?

On first go call I'm presented with OS pop-up that "macOS cannot verify that this app is free from malware", with options either to discard this window (the process is killed then) or move the binary to trash. After whitelisting this binary via Preferences → Security & Privacy → General → Allow Anyway, go can be called fine.

Next I tried to build project with go install and got the came pop-up for "compile" tool.

Screen Shot 2019-10-18 at 13 00 17

Screen Shot 2019-10-18 at 13 00 23

@artyom
Copy link
Member Author

@artyom artyom commented Oct 18, 2019

To be able to run updated Go 1.13.3 and compile my project with it, I had to whitelist so far these binaries:

  • go
  • compile
  • asm
  • cgo
  • link

Whitelisting each takes the following:

  1. presented with os pop-up about binary not being verified;
  2. click cancel, at this moment binary is killed;
  3. go to Preferences → Security & Privacy → General, click "Allow Anyway" button;
  4. try running the same command again;
  5. presented with os pop-up about binary not being verified, but now with another option to run binary anyway;
  6. click "open" button on that pop-up.
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Oct 18, 2019

Good times.

/cc @golang/osp-team

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Oct 18, 2019

Related: #34748 (we need a Catalina builder)

@networkimprov
Copy link

@networkimprov networkimprov commented Oct 18, 2019

Discussion on Hacker News: https://news.ycombinator.com/item?id=21179970

Also, Catalina is reported to be a disastrous upgrade.

@andybons
Copy link
Member

@andybons andybons commented Oct 18, 2019

I’ll take a look at this.

@andybons andybons self-assigned this Oct 18, 2019
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Oct 18, 2019

I can reproduce on my personal MBP with macOS 10.15 (19A602) when installing from the .pkg installer too, however it's a different (less intrusive) experience.

The "macOS cannot verify that this app is free from malware" prompt appears when first running the installer. After accepting and installing, Go binaries such as go, compile, do not produce the same prompt. Also, re-running the .pkg installer doesn't ask again.

@andybons
Copy link
Member

@andybons andybons commented Oct 21, 2019

In the process of submitting the installer to Apple's notary service, but the process is very slow due to a service issue on Apple's side and our internal tooling keeps timing out. Diagnosing this has been difficult.

It may be that we will need to enable the Hardened Runtime which likely requires a newer version of Xcode that we're using to build releases and an audit of whatever entitlements are required for our binaries to run.

@aclements can you take a look at the link above and let me know which you think we'll need (if any)?

@aclements
Copy link
Member

@aclements aclements commented Oct 21, 2019

@andybons, I don't think we need any of the runtime exceptions from that page. The only one that we may even potentially want is "Disable Library Validation Entitlement", which would let users use the the plugin package without signing their plugins, but it seems like developers probably should sign their plugins.

@vp2177
Copy link

@vp2177 vp2177 commented Oct 22, 2019

The (.pkg) installer doesn't even start for me. Is that the same issue, or unrelated?

Screenshot 2019-10-18 at 17 10 09

"go1.13.3.darwin-amd64.pkg can't be opened because Apple cannot check it for malicious software."

Meanwhile, the older Go version I had installed from before the OS X Catalina update, continues to work fine, no warning dialogs.

@andybons
Copy link
Member

@andybons andybons commented Oct 22, 2019

@vp2177 it’s the same issue.

@andybons
Copy link
Member

@andybons andybons commented Oct 22, 2019

Good news: according to Apple, we don’t have to enable the hardened runtime until January 2020. That said, we’ll likely have to start signing the binaries themselves (not just the installer as we do now), so this issue should stay open even if we fix the problem in the short term via notarization of the installer.

@andybons andybons pinned this issue Oct 22, 2019
@andybons andybons added the OS-Darwin label Oct 22, 2019
@andybons
Copy link
Member

@andybons andybons commented Oct 22, 2019

@dmitshur to follow up on your experience, do you have SIP (System Integrity Protection) turned on? csrutil status should give you that info.

@artyom do you have SIP turned on? Thanks.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Oct 22, 2019

@andybons Yes.

$ csrutil status
System Integrity Protection status: enabled.
@artyom
Copy link
Member Author

@artyom artyom commented Oct 22, 2019

@andybons yes, it's enabled:

¶ csrutil status
System Integrity Protection status: enabled.
@andybons
Copy link
Member

@andybons andybons commented Oct 22, 2019

Our installer is getting stuck in limbo with the Notarization service. We’ve contacted Apple and are awaiting a response. It’s unclear what the ETA on that will be, though.

@andybons
Copy link
Member

@andybons andybons commented Oct 22, 2019

https://developer.apple.com/system-status/ now shows an issue with the Notary service:

Performance: Today, 10:10 AM - ongoing

Some users are affected

Users are experiencing a problem with this service. We are working to resolve this issue.

@andybons andybons changed the title Go 1.13.3 distribution lacks notarization for macOS Catalina, requiring whitelisting of each binary Notarize our installer and binaries (required as of macOS 10.15 Catalina) Oct 22, 2019
@andybons
Copy link
Member

@andybons andybons commented Oct 23, 2019

Notarization of the installer package has completed successfully. We are testing locally to ensure no scary popups are shown.

@andybons
Copy link
Member

@andybons andybons commented Oct 23, 2019

While we believe we have the installer package issue resolved, we have to deal with the .tar.gz bundles, which require notarization of each individual binary. Working on this.

@rsc rsc changed the title Notarize our installer and binaries (required as of macOS 10.15 Catalina) macOS 10.15 Catalina rejects non-notarized Go installer, binaries Oct 24, 2019
@andybons
Copy link
Member

@andybons andybons commented Oct 24, 2019

As an update, this is what we are going to do:

  • Sign all Mach-O binaries within $GOROOT/bin/ and $GOROOT/pkg/tool/darwin_amd64 (without the Hardened Runtime enabled as it’s not required just yet)
  • Construct the .pkg installer with the signed binaries included
  • Sign the .pkg installer
  • Send the .pkg installer off to the notarization service and staple the resulting ticket to the installer

Since the notarization process is recursive (and the bits being distributed in the .tar.gz file are the same that will be in the installer), this should solve the warnings for users that choose to install using either the .pkg file or the .tar.gz file.

I will update this thread once this work is complete.

@andybons
Copy link
Member

@andybons andybons commented Oct 29, 2019

The initial work mentioned in #34986 (comment) is now complete. We will likely issue a point release this week with the changes after some further testing.

@rsc rsc unpinned this issue Oct 30, 2019
@rsc rsc pinned this issue Oct 30, 2019
gopherbot pushed a commit that referenced this issue Nov 21, 2019
Starting with macOS 10.15 (Catalina), Apple now requires all software
distributed outside of the App Store to be notarized. Any binaries we
distribute must abide by a strict set of requirements like code-signing
and having a minimum target SDK of 10.9 (amongst others).

Apple’s notarization service will recursively inspect archives looking to
find notarization candidate binaries. If it finds a binary that does not
meet the requirements or is unable to decompress an archive, it will
reject the entire distribution. From cursory testing, it seems that the
service uses content sniffing to determine file types, so changing
the file extension will not work.

There are some binaries and archives included in our distribution that
are being detected by Apple’s service as potential candidates for
notarization or decompression. As these are files used by tests and some
are intentionally invalid, we don’t intend to ever make them compliant.

As a workaround for this, we base64-encode any binaries or archives that
Apple’s notarization service issues a warning for, as these warnings will
become errors in January 2020.

Updates #34986

Change-Id: I106fbb6227b61eb221755568f047ee11103c1680
Reviewed-on: https://go-review.googlesource.com/c/go/+/208118
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 21, 2019

Change https://golang.org/cl/208219 mentions this issue: [release-branch.go1.13] all: base64-encode binaries that will cause Apple notarization to fail

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 21, 2019

Change https://golang.org/cl/208220 mentions this issue: [release-branch.go1.12] all: base64-encode binaries that will cause Apple notarization to fail

@andybons
Copy link
Member

@andybons andybons commented Nov 21, 2019

@gopherbot please open a backport for 1.12 and 1.13 since otherwise Apple will reject future point releases of those versions

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 21, 2019

Backport issue(s) opened: #35747 (for 1.12), #35748 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

1 similar comment
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 21, 2019

Backport issue(s) opened: #35747 (for 1.12), #35748 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@nightlyone
Copy link
Contributor

@nightlyone nightlyone commented Nov 21, 2019

I wonder why tests and test data need to be shipped in releases. People interested in that can always retrieve it from the source repository. Has that been considered?

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Nov 21, 2019

@nightlyone, it has, several times. The main reason we continue to include them is that when a user reports a problem, it's nice to be able to ask them "What does go test syscall or go test -v os say?". That's a question they can easily answer. If we make them do a new download, build, and parallel install, they're very unlikely to do so. Admittedly this debugging procedure doesn't happen too often, though.

gopherbot pushed a commit that referenced this issue Nov 21, 2019
…pple notarization to fail

Starting with macOS 10.15 (Catalina), Apple now requires all software
distributed outside of the App Store to be notarized. Any binaries we
distribute must abide by a strict set of requirements like code-signing
and having a minimum target SDK of 10.9 (amongst others).

Apple’s notarization service will recursively inspect archives looking to
find notarization candidate binaries. If it finds a binary that does not
meet the requirements or is unable to decompress an archive, it will
reject the entire distribution. From cursory testing, it seems that the
service uses content sniffing to determine file types, so changing
the file extension will not work.

There are some binaries and archives included in our distribution that
are being detected by Apple’s service as potential candidates for
notarization or decompression. As these are files used by tests and some
are intentionally invalid, we don’t intend to ever make them compliant.

As a workaround for this, we base64-encode any binaries or archives that
Apple’s notarization service issues a warning for, as these warnings will
become errors in January 2020.

Updates #34986
Fixes #35748

Change-Id: I106fbb6227b61eb221755568f047ee11103c1680
Reviewed-on: https://go-review.googlesource.com/c/go/+/208118
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
(cherry picked from commit 8bbfc51)
Reviewed-on: https://go-review.googlesource.com/c/go/+/208219
Reviewed-by: Alexander Rakoczy <alex@golang.org>
gopherbot pushed a commit that referenced this issue Nov 21, 2019
…pple notarization to fail

Starting with macOS 10.15 (Catalina), Apple now requires all software
distributed outside of the App Store to be notarized. Any binaries we
distribute must abide by a strict set of requirements like code-signing
and having a minimum target SDK of 10.9 (amongst others).

Apple’s notarization service will recursively inspect archives looking to
find notarization candidate binaries. If it finds a binary that does not
meet the requirements or is unable to decompress an archive, it will
reject the entire distribution. From cursory testing, it seems that the
service uses content sniffing to determine file types, so changing
the file extension will not work.

There are some binaries and archives included in our distribution that
are being detected by Apple’s service as potential candidates for
notarization or decompression. As these are files used by tests and some
are intentionally invalid, we don’t intend to ever make them compliant.

As a workaround for this, we base64-encode any binaries or archives that
Apple’s notarization service issues a warning for, as these warnings will
become errors in January 2020.

Updates #34986
Updates #35747

Change-Id: I106fbb6227b61eb221755568f047ee11103c1680
Reviewed-on: https://go-review.googlesource.com/c/go/+/208118
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
(cherry picked from commit 8bbfc51)
Reviewed-on: https://go-review.googlesource.com/c/go/+/208220
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@bcmills
Copy link
Member

@bcmills bcmills commented Nov 21, 2019

It also seems desirable for go test all to have the same behavior regardless of how the Go toolchain was installed, particularly in module mode (where all is actually a reasonable thing to want to test).

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 21, 2019

Change https://golang.org/cl/208227 mentions this issue: [release-branch.go1.12] cmd/vendor: remove _test.go and testdata files

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 22, 2019

Change https://golang.org/cl/208266 mentions this issue: cmd/release, cmd/releasebot: clean up unused code

gopherbot pushed a commit to golang/build that referenced this issue Nov 22, 2019
+ The .pkg installer is now constructed by an internal process,
  since the binaries within the installer need to be signed using
  internal-only certs
+ 1.11 is no longer supported, so the tour is no longer shipped
  with the release

Updates golang/go#34986

Change-Id: Ic05198dec2fdbfb26d9011944051a97c777e3898
Reviewed-on: https://go-review.googlesource.com/c/build/+/208266
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
gopherbot pushed a commit that referenced this issue Nov 22, 2019
Binary files included in testdata directories can cause Apple’s
notarization service to reject us since they don’t abide by their
strict requirements.

To emulate go mod vendor, remove all _test.go and testdata files
from the vendor directory and update the instructions.

Updates #34986
Fixes #35747

Change-Id: I5cde905fc78838d2e3b1519dab4aeee13d8d5356
Reviewed-on: https://go-review.googlesource.com/c/go/+/208227
Run-TryBot: Andrew Bonventre <andybons@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@andybons
Copy link
Member

@andybons andybons commented Nov 26, 2019

I've run an altered all.bash that signs (with the hardened runtime enabled) the toolchain binaries and all tests passed on macOS 10.15.1.

We plan to enable the hardened runtime for the 1.14 beta (scheduled for early December) and see if anything breaks. If everything looks good, we'll enable it for all releases moving forward.

@toothrot
Copy link
Contributor

@toothrot toothrot commented Dec 16, 2019

Just to reiterate what Andy said, the hardened runtime will be enabled for the go1.14 beta release.

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Jan 30, 2020

I've confirmed with @bcmills that #36572 should not be considered a blocker for go1.14rc1. I'm going to recommend that we close this issue. @andybons @dmitshur @toothrot

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Feb 4, 2020

Sounds good. With the hardened runtime enabled as of Go 1.14 beta 1, I don't believe there's anything left to do here other than to watch for any issues if they're reported, but new issues can be opened for that.

I'll let one of @andybons or @toothrot confirm and close this issue since they're currently assigned to it.

@andybons
Copy link
Member

@andybons andybons commented Feb 4, 2020

Yes this looks to be fixed. If we discover any issues during the notarization stage of a release we'll open a new issue.

@andybons andybons closed this Feb 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.