Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: add option to create multi-architecture fat binaries on Darwin #40698

Open
jtsylve opened this issue Aug 11, 2020 · 18 comments
Open

cmd/go: add option to create multi-architecture fat binaries on Darwin #40698

jtsylve opened this issue Aug 11, 2020 · 18 comments

Comments

@jtsylve
Copy link
Contributor

@jtsylve jtsylve commented Aug 11, 2020

Given the Apple silicon announcement, it will soon be common for developers to have to support both arm64le and x86_64 variants of macOS. Rather than shipping multiple binaries, it would be nice to be able to ship macho fat binaries that contain both implementations without having to manually use tools like lipo. Could multi-architecture support be added to the compiler?

@ianlancetaylor ianlancetaylor changed the title Feature Request: macho: add option to create multi-architecture fat binaries cmd/go: add option to create multi-architecture fat binaries on Darwin Aug 11, 2020
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Aug 11, 2020
@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 11, 2020

My first guess would be that this would involve the go tool invoking the compiler once for each architecture, and then invoking some tool to package the resulting executables into a fat binary.

CC @bcmills @jayconrod

@rsc
Copy link
Contributor

@rsc rsc commented Aug 11, 2020

I don't see this happening in the go command in the short term. It would be a major change to juggle multiple architectures during a single command invocation. We've got a lot of modules work left to do for Go 1.16.

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 12, 2020

It needn't be a feature of go build. A separate tool would be most helpful.

I build & assemble release tarballs for MacOS, Windows, and Linux with a shell script on Linux. That will break down for MacOS without a cross-platform command to create a fat binary.

cc @eliasnaur @hyangah @hajimehoshi

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 12, 2020

My understanding is that given two ordinary binaries, you can create a fat binary using the lipo command.

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 12, 2020

EDIT: Is there a lipo for Linux? I haven't found a man page for one.

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 12, 2020

This is about Darwin, not Linux.

https://ss64.com/osx/lipo.html

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 12, 2020

Thanks, but that doesn't help. See #40698 (comment) for why I asked about Linux.

@randall77
Copy link
Contributor

@randall77 randall77 commented Aug 12, 2020

lipo is open source, so it can be ported. Here's one attempt: https://github.com/hogliux/cctools (no endorsement implied, YMMV).
It would be great if a lipo tool were written in Go so it could port easily. You could just go get it. I would imagine a lipo tool would be really easy to write, but I haven't investigated deeply.
Regardless, this doesn't have to be in the go tool, at least to start. A go-gettable tool should be fine.

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 12, 2020

I'd need to know that the authors of a utility that creates binaries for distribution to end users
a) aren't malicious,
b) know what they're doing, and
c) will promptly post fixes if any security issues arise.

The only quick way to vet some-random-lipo is to compare its output with MacOS lipo, and then we're back on square zero.

@randall77
Copy link
Contributor

@randall77 randall77 commented Aug 12, 2020

@networkimprov Sure, but that same problem applies to any utility. We can't put the universe in the Go distribution.
Go has to draw the line somewhere as to what we want to take on with regards to your 3 points. It's not free to take on those responsibilities for a new binary.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Aug 12, 2020

The fat binary format is simple. You can see the relevant definitions at https://opensource.apple.com/source/xnu/xnu-344.49/EXTERNAL_HEADERS/mach-o/fat.h.auto.html. The file starts with a fat_header. There are then nfat_arch instances of fat_arch. See the comment in the file. It should only take a couple of hours to write that in a Go program.

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 12, 2020

A lipo tool doesn't have to land in Go 1.16; the x/ tree is fine for now.

Ian, thanks for the feasibility assessment.

@randall77
Copy link
Contributor

@randall77 randall77 commented Aug 24, 2020

There's a more recent fat.h than the one Ian linked to. It handles files larger than 4GB, among other things.

If you're on a Mac, the header file is at /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/mach-o/fat.h

A good reference: https://stackoverflow.com/questions/46965686/parsing-universal-fat-binary-files

@randall77
Copy link
Contributor

@randall77 randall77 commented Aug 24, 2020

I hacked up a tool that can make fat binaries: https://github.com/randall77/makefat

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 24, 2020

\o/ thank you!

Readme needs a command line summary.

And it could use a test script to build a hello-world app, run makefat, then launch. Would you accept a patch for that?

@randall77
Copy link
Contributor

@randall77 randall77 commented Aug 24, 2020

Sure.

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 25, 2020

I verified my script with GOARCH=386 & =amd64. What should it use for Apple Silicon?
With 1.13.3 on Linux, =arm64 doesn't work:

$ GOOS=darwin GOARCH=arm64 go build -o "$test-arm64" "$test.go"
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: unrecognized option '-pagezero_size'
/usr/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status

Also, makefat.go needn't panic if the command-line arguments are incorrect, os.Exit(1) would be fine :-)

@cherrymui
Copy link
Contributor

@cherrymui cherrymui commented Aug 25, 2020

arm64

The Apple Silicon port doesn't work yet. I'm working on it.

@rsc rsc modified the milestones: Backlog, Unplanned Sep 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.