Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: reproducible builds don't work on macOS #40979

Closed
mandrigin opened this issue Aug 22, 2020 · 13 comments
Closed

cmd/link: reproducible builds don't work on macOS #40979

mandrigin opened this issue Aug 22, 2020 · 13 comments
Milestone

Comments

@mandrigin
Copy link

@mandrigin mandrigin commented Aug 22, 2020

What version of Go are you using (go version)?

$ go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/mandrigin/Library/Caches/go-build"
GOENV="/Users/mandrigin/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/mandrigin/golang/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/mandrigin/golang"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/mandrigin/Desktop/repro-builds-issue/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/y5/rmr800b53s53zt42d2tfmpdc0000gn/T/go-build570372631=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I've created a repository with a small app where I can reproduce this issue easily. It uses one module with native files (go-duktape): https://github.com/mandrigin/golang-reproducible-builds-issue-osx/

The app builds the same source twice into 2 different files
go build -asmflags -trimpath -ldflags=-buildid= -o t1 .
go build -asmflags -trimpath -ldflags=-buildid= -o t2 .

Then it takes sha sums of both files and prints them. Also it prints a diff between these files.

What did you expect to see?

Reproducible builds on both Alpine (in Docker) and macOS.

What did you see instead?

Reproducible builds on Alpine work. (hashes match, no diff between binary files).

Reproducible builds on macOS are broken.
Diff on macOS contains entries like this one

< 005b1850: 6e2f 542f 676f 2d6c 696e 6b2d 3732 3938  n/T/go-link-7298
< 005b1860: 3738 3633 362f 3030 3030 3237 2e6f 005f  78636/000027.o._
---
> 005b1850: 6e2f 542f 676f 2d6c 696e 6b2d 3532 3038  n/T/go-link-5208
> 005b1860: 3330 3433 312f 3030 3030 3237 2e6f 005f  30431/000027.o._

Clang Version

Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
@ianlancetaylor ianlancetaylor changed the title CGO: reproducible builds don't work on macOS cmd/link: reproducible builds don't work on macOS Aug 22, 2020
@ianlancetaylor ianlancetaylor added this to the Go1.16 milestone Aug 22, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 22, 2020

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 23, 2020

It looks like temporary file names are built into the binary. I'll take a look.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 23, 2020

Yeah, it seems the case for any cgo program on darwin, since at least Go 1.11 (Go 1.10 doesn't work on my machine).

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 23, 2020

I think it is the darwin linker adds them. I don't know how to make it not to...

@thanm
Copy link
Member

@thanm thanm commented Aug 23, 2020

Possibly DWARF related? What happens when "-w" is added to linker options?

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 23, 2020

-w doesn't help.

If I understand the otool dump correctly, the file paths are in __LINKEDIT segment.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 24, 2020

Are the paths coming from the command line? Would it help to chdir to the directory, and just pass plain .o files to the linker?

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 24, 2020

Good point. The command line does include the paths. But if I cd into the temp dir and run the command with only the file name without the directory, the binary still contains the paths...

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 24, 2020

If you chdir, does it record the paths for each object, or is it recording the current directory somewhere?

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 24, 2020

The paths for each object.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 24, 2020

Well that's quite annoying.

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 26, 2020

Change https://golang.org/cl/250944 mentions this issue: cmd/link: strip STAB (symbolic debugging) symbols on darwin

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 4, 2020

Change https://golang.org/cl/253157 mentions this issue: cmd/link: add a test for reproducible build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.