Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/mobile: gomobile bind is ignoring target architecture parameter #47212

Open
iamcalledrob opened this issue Jul 14, 2021 · 34 comments
Open

x/mobile: gomobile bind is ignoring target architecture parameter #47212

iamcalledrob opened this issue Jul 14, 2021 · 34 comments

Comments

@iamcalledrob
Copy link

@iamcalledrob iamcalledrob commented Jul 14, 2021

What version of Go are you using (go version)?

$ go version
go version go1.16.5 darwin/arm64

Does this issue reproduce with the latest release?

Yes (and only with the latest release)

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/rob/Library/Caches/go-build"
GOENV="/Users/rob/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/rob/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/rob/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.16.5"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/3l/mj1p0_6x3dnb850mkd9ph8b00000gn/T/go-build3577254656=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I ran gomobile bind with a specific target and architecture (amd64 in this example)

Exact bind command:

gomobile bind -target ios/amd64 -v -o mobile.xcframework ./pkg/...

Gomobile version

golang.org/x/mobile v0.0.0-20210710064935-76c259c465ba

What did you expect to see?

I expected a framework to be built only for iOS with the amd64 architecture (i.e. for an intel simulator). I would expect to see something like this inside the xcframework:

$ ls -l mobile.xcframework 
drwxr-xr-x  4 rob  staff   128 Jul 14 15:55 ios-x86_64-simulator

What did you see instead?

The framework was built for all "iOS" platforms (macos, catalyst, simulator, iOS) and architectures (amd64, arm64).

$ ls -l mobile.xcframework 
-rw-r--r--  1 rob  staff  1767 Jul 14 15:54 Info.plist
drwxr-xr-x  4 rob  staff   128 Jul 14 15:59 ios-arm64
drwxr-xr-x  3 rob  staff    96 Jul 14 15:54 ios-arm64_x86_64-maccatalyst
drwxr-xr-x  4 rob  staff   128 Jul 14 15:55 ios-arm64_x86_64-simulator
drwxr-xr-x  4 rob  staff   128 Jul 14 15:58 macos-arm64_x86_64

The additional platforms are a result of the latest release (golang/mobile@76c259c).

I think this is a combination of both a bug and some missing functionality—i.e. being able to specify which platforms to build for in addition to which architectures.

There's a comment by another person on this release with more information on a specific bug regarding this: golang/mobile@76c259c#commitcomment-53412472

@iamcalledrob
Copy link
Author

@iamcalledrob iamcalledrob commented Jul 14, 2021

More context: It looks like a PR was made whilst I was writing this issue (!) which might include fixes for the bug and missing functionality golang/mobile#70

@gopherbot gopherbot added this to the Unreleased milestone Jul 14, 2021
@ydnar
Copy link

@ydnar ydnar commented Jul 15, 2021

@iamcalledrob hi there, golang/mobile#70 / http://golang.org/cl/334689 author here.

It turns out that while golang/mobile#65 worked for some use cases, it subtly broke others. Sorry!

@hajimehoshi asked if I could file an issue to discuss the changes in the new PR, but it turns out someone (you) already did.

So to summarize what I think the problem is, and some thoughts on how we might fix it:

  • golang.org/x/mobile predates Go modules, GOOS=ios, and currently has some outdated defaults that aren’t supported by current versions of Xcode.
  • golang/mobile#65 and its antecedents overloaded -target=ios to mean ios, iphonesimulator, macosx, and maccatalyst SDKs.
  • Go 1.16 introduced GOOS=ios with a darwin build tag: #42100
  • gomobile wasn’t aware of GOOS=ios, and continued to build with GOOS=darwin and an ios build tag. When combined with building additional SDKs (macosx, maccatalyst), subtle breakage could occur.

My goal with golang/mobile#70 was to address these subtle breakages, and update gomobile to support GOOS=ios as well as be able to correctly build for the user’s choice of platforms, rather than forcing a macos and maccatalyst build on every user.

  • The new PR restores the functionality of gomobile -target=ios to build just the ios and iphonesimulator platforms.
  • If you want all supported Apple platforms, use -target=darwin. I’m open to nuking this. @hajimehoshi what do you think?
  • It expands the current list of supported platforms of android and ios, adding macos, macatalyst (UIKit on macOS), iossimulator. These can all be individually targeted, along with a specific arch if you want, e.g. iossimulator/amd64.
  • It adds a handful of new build tags so Go code can discriminate between the different Apple platforms. The new build tags are macos, maccatalyst, and iossimulator. For instance, you can use the maccatalyst tag to build a Go file that accesses UIKit.

@iamcalledrob could you try your project with this branch? -target=ios/amd64 should do what you want now. Add this to your go.mod file and run go mod tidy:

replace golang.org/x/mobile => github.com/ydnar/gomobile goos-ios

@ydnar
Copy link

@ydnar ydnar commented Jul 15, 2021

@hajimehoshi does the mobile repo have a macOS trybot?

@hajimehoshi
Copy link
Member

@hajimehoshi hajimehoshi commented Jul 15, 2021

CC @hyangah

The issue that a specified architecture is ignored (as the first comment says) is a regression. Also, we have multiple issues as @ydnar listed. I think we should revert the Catalyst change once, and then re-discuss the design. @ydnar What do you think?

If you want all supported Apple platforms, use -target=darwin. I’m open to nuking this. @hajimehoshi what do you think?

This seems good, as macOS is not iOS, but let me think more...

@hajimehoshi does the mobile repo have a macOS trybot?

I don't know about this, but I don't think there is.

@ydnar
Copy link

@ydnar ydnar commented Jul 15, 2021

The issue that a specified architecture is ignored (as the first comment says) is a regression. Also, we have multiple issues as @ydnar listed. I think we should revert the Catalyst change once, and then re-discuss the design. @ydnar What do you think?

I’m OK reverting the change, and revising it on the new PR, since it preserves the earlier behavior of -target=ios building just for iOS and the simulator. That’s better than forcing undesired platforms (and any related build issues) on gomobile’s users.

If you want all supported Apple platforms, use -target=darwin. I’m open to nuking this. @hajimehoshi what do you think?

This seems good, as macOS is not iOS, but let me think more...

The alternative would be for developers to manually specify all the platforms they want to build for, e.g. -target=ios,macos, which seems OK.

@hajimehoshi does the mobile repo have a macOS trybot?

I don't know about this, but I don't think there is.

Given that this needs to run tests on macOS with Xcode (and a developer environment with a cert), it’s probably a good idea. That would have caught a few of the broken tests, some of which predate the Catalyst PR. Is it possible to add one?

@hajimehoshi
Copy link
Member

@hajimehoshi hajimehoshi commented Jul 15, 2021

Given that this needs to run tests on macOS with Xcode (and a developer environment with a cert), it’s probably a good idea. That would have caught a few of the broken tests, some of which predate the Catalyst PR. Is it possible to add one?

CC @dmitshur

@gopherbot
Copy link

@gopherbot gopherbot commented Jul 15, 2021

Change https://golang.org/cl/334590 mentions this issue: Revert "cmd/gomobile: support macOS and Catalyst"

@hajimehoshi
Copy link
Member

@hajimehoshi hajimehoshi commented Jul 15, 2021

gomobile wasn’t aware of GOOS=ios, and continued to build with GOOS=darwin and an ios build tag. When combined with building additional SDKs (macosx, maccatalyst), subtle breakage could occur.

I'd like to know whether there are things Go should care for macOS Catalyst, like what Go does for iOS. Or, is it enough for Go to behave as if the OS is macOS or iOS for macOS Catalyst?

https://github.com/golang/go/search?q=filename%3A*_ios.go&type=code

@ydnar
Copy link

@ydnar ydnar commented Jul 15, 2021

I found an issue in cgo that’s potentially blocking a real fix for this: #47228

@ydnar
Copy link

@ydnar ydnar commented Jul 15, 2021

@hajimehoshi my current workaround for #47228 is to supply both ios and macos build tags when building for Catalyst.

@ydnar
Copy link

@ydnar ydnar commented Jul 15, 2021

If you want all supported Apple platforms, use -target=darwin. I’m open to nuking this. @hajimehoshi what do you think?

This seems good, as macOS is not iOS, but let me think more...

The alternative would be for developers to manually specify all the platforms they want to build for, e.g. -target=ios,macos, which seems OK.

I removed target=darwin and target=iossimulator in current build of https://go-review.googlesource.com/c/mobile/+/334689

It also fixes OP’s bug, and builds with only a single target arch will now succeed.

gopherbot pushed a commit to golang/mobile that referenced this issue Jul 16, 2021
This reverts commit 76c259c.

Reason for revert: Regression. Specifying architectures doesn't work. See golang/go#47212

Change-Id: I3200316cf28535cfb48e37636bc3b9d14d13e91e
Reviewed-on: https://go-review.googlesource.com/c/mobile/+/334590
Trust: Hajime Hoshi <hajimehoshi@gmail.com>
Trust: Hyang-Ah Hana Kim <hyangah@gmail.com>
Run-TryBot: Hajime Hoshi <hajimehoshi@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
@iamcalledrob
Copy link
Author

@iamcalledrob iamcalledrob commented Jul 16, 2021

@iamcalledrob could you try your project with this branch? -target=ios/amd64 should do what you want now. Add this to your go.mod file and run go mod tidy:

replace golang.org/x/mobile => github.com/ydnar/gomobile goos-ios

I can confirm that things are working perfectly with your new branch (amazing!). The biggest benefit here is the build times while developing—since a build can now be for a specific arch/platform, the build time is reduced from ~90s to ~13s.

For anyone else who might be following this, I had to also go get golang.org/x/mobile/cmd/gomobile to pick up the new tools :)

@iamcalledrob
Copy link
Author

@iamcalledrob iamcalledrob commented Jul 16, 2021

Following up, I'm unable to get builds with -target ios or -target ios/arm64 to work in the iOS simulator on my M1 mac.

From peeking at the code, it looks like there's an assumption made that ios/arm64 is a physical device, and ios/amd64 is simulator. M1 complicates that assumption a bit :)

@ydnar
Copy link

@ydnar ydnar commented Jul 16, 2021

Following up, I'm unable to get builds with -target ios or -target ios/arm64 to work in the iOS simulator on my M1 mac.

From peeking at the code, it looks like there's an assumption made that ios/arm64 is a physical device, and ios/amd64 is simulator. M1 complicates that assumption a bit :)

Curious, are you able to build for iOS simulator with current master branch of golang/mobile?

I ran into an issue merging ios/arm64 and iphonesimulator/arm64 frameworks into a single XCFramework. Maybe the solution I chose (inhibit creating an iphonesimulator/arm64 slice) wasn’t right.

@iamcalledrob
Copy link
Author

@iamcalledrob iamcalledrob commented Jul 16, 2021

I'm also not able to build for iOS simulator with the master branch: building for iOS Simulator, but linking in object file built for iOS, for architecture arm64. It appears to be this problem: #45805?

When building from the github.com/ydnar/gomobile goos-ios branch, the error is a little different (missing arm64 architecture for iphonesimulator):
ld: warning: ignoring file /Users/rob/Library/Developer/{...}/Debug-iphonesimulator/Xplat.framework/Xplat, missing required architecture arm64 in file /Users/rob/Library/Developer/{...}Debug-iphonesimulator/Xplat.framework/Xplat (1 slices)

@ydnar
Copy link

@ydnar ydnar commented Jul 16, 2021

I think I got it! Going to run a couple more tests and push a commit:

Screen Shot 2021-07-16 at 4 04 26 PM

Edit: commit pushed. @iamcalledrob can you try with current HEAD?

  • It re-adds an iossimulator platform (name TBD), which has its own configuration.
  • The ios platform can only be built for arm64. If you try to build -target=ios/amd64 it will fail. You’ll need to supply -target=iossimulator/amd64 to get an x86_64 slice for the simulator.
  • If you run gomobile build -target=ios it will implicitly expand ios to ios,iossimulator . This probably isn’t the right design, but it works.

Edit: building for iOS, iPhoneSimulator, macCatalyst, and macOS with arm64 and amd64 architectures in the same XCFramework now work. I’ve tested this with a simple app in a simulator:

Screen Shot 2021-07-16 at 4 28 00 PM

ydnar added a commit to ydnar/gomobile that referenced this issue Jul 16, 2021
This enables building for iphonesimulator-arm64 and merging into a fat
XCFramework that also may include ios/arm64.

This should fix golang/go#47212 (comment)

Note: tests are broken. Will fix in a followup commit.
@iamcalledrob
Copy link
Author

@iamcalledrob iamcalledrob commented Jul 16, 2021

Brilliant. I can confirm this now builds a working framework for the ios simulator on arm64!

@ydnar
Copy link

@ydnar ydnar commented Jul 17, 2021

Brilliant. I can confirm this now builds a working framework for the ios simulator on arm64!

Terrific. I just pushed a couple updates. Target iphonesimulator is now iossimulator.

@Milerius
Copy link

@Milerius Milerius commented Jul 21, 2021

Nice work @ydnar i hope it's will be merged soon <3

@ydnar
Copy link

@ydnar ydnar commented Jul 21, 2021

@Milerius me too!

@hajimehoshi what do you think about reviewing the updated CL?

@hajimehoshi
Copy link
Member

@hajimehoshi hajimehoshi commented Jul 21, 2021

My understanding is that Go itself doesn't support Catalyst yet, then we should fix Go first. Is that correct?

@ydnar
Copy link

@ydnar ydnar commented Jul 21, 2021

My understanding is that Go itself doesn't support Catalyst yet, then we should fix Go first. Is that correct?

This CL doesn’t require any upstream fixes to Go to land it. gomobile bind works for iOS + multiple platforms + architectures.

It’s possible to improve this later if Go is Catalyst-aware, potentially supporting gomobile build for a Catalyst target. My sense is the primary use case for x/mobile is creating libraries, rather than building whole apps.

@hajimehoshi
Copy link
Member

@hajimehoshi hajimehoshi commented Jul 21, 2021

Oh sure. I'll take a look.

Which CL are you talking about? https://go-review.googlesource.com/c/mobile/+/334689 still has Catalyst-related code.

@ydnar
Copy link

@ydnar ydnar commented Jul 21, 2021

Oh sure. I'll take a look.

Which CL are you talking about? https://go-review.googlesource.com/c/mobile/+/334689 still has Catalyst-related code.

Yes, that one. Catalyst support is kind of the point.

  • It can build for iOS and Simulator, and has a backwards-compatible interface with current master branch.
  • It can build fat XCFrameworks for multiple SDKs and architectures.
  • Catalyst support is a bonus, and from what I can tell, folks really want that feature. Its inclusion doesn’t break anything in the other supported platforms. Unlike my previous reverted CL, Catalyst support is entirely optional. Users have to request it specifically by specifying -target=maccatalyst.

@hajimehoshi
Copy link
Member

@hajimehoshi hajimehoshi commented Jul 21, 2021

As it is still impossible to support Catalyst by Go now, we should remove the code for Catalyst from the CL and leave TODO comments. Also please update the description of the CL. Let's revisit supporting Catalyst later.

@ydnar
Copy link

@ydnar ydnar commented Jul 21, 2021

As it is still impossible to support Catalyst by Go now, we should remove the code for Catalyst from the CL and leave TODO comments. Also please update the description of the CL. Let's revisit supporting Catalyst later.

It’s not impossible. A Catalyst framework generated by gomobile in this CL works in a Catalyst app. Do you need an example?

@hajimehoshi
Copy link
Member

@hajimehoshi hajimehoshi commented Jul 21, 2021

I think it works in most cases but is not well tested by the Go team. The Go itself has iOS-specific code in the standard libraries, and we're not sure how Catalyst code should treat them. We might need more fix like Catalyst-specific code.

@ydnar
Copy link

@ydnar ydnar commented Jul 21, 2021

I see your point. Given that x/mobile is outside of the Go core, and not subject to the same compatibility guarantee, what about marking Catalyst support as experimental?

The Go Mobile project is experimental. Use this at your own risk. While we are working hard to improve it, neither Google nor the Go team can provide end-user support.

There are 18 lines (10 SLOC) that mention "catalyst" in this CL, and it doesn’t interfere with the other platforms:

$ grep -i catalyst ./cmd/gomobile/*
./cmd/gomobile/bind_darwinapp.go:		// Catalyst support requires iOS 13+
./cmd/gomobile/bind_darwinapp.go:		if platform == "maccatalyst" && v < 13.0 {
./cmd/gomobile/bind_darwinapp.go:			return errors.New("catalyst requires -iosversion=13 or higher")
./cmd/gomobile/build.go:				// Catalyst support requires iOS 13+
./cmd/gomobile/build.go:				if platform == "maccatalyst" && v < 13.0 {
./cmd/gomobile/build.go:					return nil, errors.New("catalyst requires -iosversion=13 or higher")
./cmd/gomobile/build.go://    ios,iossimulator,maccatalyst
./cmd/gomobile/doc.go:	gomobile bind [-target android|ios|iossimulator|macos|maccatalyst] [-bootclasspath <path>] [-classpath <path>] [-o output] [build flags] [package]
./cmd/gomobile/doc.go:either android (the default) or ios, iossimulator, macos, maccatalyst.
./cmd/gomobile/doc.go:	gomobile build [-target android|ios|iossimulator|macos|maccatalyst] [-o output] [-bundleid bundleID] [build flags] [package]
./cmd/gomobile/doc.go:default) or ios, iossimulator, macos, maccatalyst.
./cmd/gomobile/env.go:var darwinPlatforms = []string{"ios", "iossimulator", "macos", "maccatalyst"}
./cmd/gomobile/env.go:	case "macos", "maccatalyst":
./cmd/gomobile/env.go:	case "macos", "maccatalyst":
./cmd/gomobile/env.go:	case "maccatalyst":
./cmd/gomobile/env.go:		// TODO(ydnar): remove tag "ios" when cgo supports Catalyst
./cmd/gomobile/env.go:		return []string{"ios", "macos", "maccatalyst"}
./cmd/gomobile/env.go:			case "maccatalyst":

In summary, I’m happy to split this CL into two parts if you think it’s necessary:

  1. Fat XCFrameworks, macOS support, test fixes
  2. +18 lines for Catalyst support

@hajimehoshi
Copy link
Member

@hajimehoshi hajimehoshi commented Jul 21, 2021

I see your point. Given that x/mobile is outside of the Go core, and not subject to the same compatibility guarantee, what about marking Catalyst support as experimental?

I slightly tend to against it, but the Go team might allow this. CC @hyangah

@nadimkobeissi
Copy link

@nadimkobeissi nadimkobeissi commented Jul 27, 2021

I strongly support @ydnar's proposed fix here.

@nadimkobeissi
Copy link

@nadimkobeissi nadimkobeissi commented Aug 21, 2021

@hajimehoshi @hyangah @ydnar What's the status on this? :-)

@ydnar
Copy link

@ydnar ydnar commented Aug 21, 2021

@nadimkobeissi

@hajimehoshi @hyangah @ydnar What's the status on this? :-)

You can use this branch now with a replace directive in your go.mod file. Otherwise I think we’re waiting for a response from @hyangah.

@hyangah
Copy link
Contributor

@hyangah hyangah commented Sep 8, 2021

This whole project is highly experimental - so it's fine to include the catalyst handling code IMO.

@ydnar
Copy link

@ydnar ydnar commented Sep 17, 2021

@nadimkobeissi @iamcalledrob golang.org/cl/334689 was merged, so Catalyst and macOS support should be working again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants