x/mobile/cmd/gomobile: multiple packages binding

[hyangah]

I need to gobind multiple packages for an app.

I considered running gomobile for each package and import multiple .so files, but that will fail due to various reasons:

• gobind framework for java assumes a single 'gojni.so' file
• duplicate code for go runtime in multiple .so files; conflicting symbols, storage, ...

Option 1:

gomobile bind -target=android -o=myapp.aar pkg1 pkg2 pkg3

The command will generate separate java classes for those packages, and gather all asset files under one asset directory. Conflicting package names and asset file names are error.

• Problem of proposal 1
  I was about to implement fix for x/mobile/cmd/gobind: Allow custom java package prefix #9660 and I don't know how it works with multiple packages.

Option 1.a

gomobile bind -target=android -o=myapp.aar -pkg=com.example.foo,org.example.bar,net.example.baz pkg pkg2 pkg3

Option 1.b

gomobile bind -target=android -o=myapp.aar pkg1:com.example.foo pkg2:org.example.bar pkg3 ...

Option 2:

gomobile bind -target=android -o=myapp.aar -list=something.json

Make gomobile take an input file of a certain format that lists all packages for gobind.

   /* something.json */
   {
        Name: "pkg1",
        Android: "com.example.foo",
        IOS: "MyFoo"
   }

Option 3:

Fake main package file - so when gomobile bind sees the 'main' package, it runs gobind on all the imported packages, and process the command that specifies the custom package name for java and prefix for obj-c.

gomobile bind -target=android -o myapp.aar myapp.go

   /* myapp.go */
   package main
   import (
          _ "pkg1"  // android:com.example.foo ios:MyFoo
          ...
   )
   func main() {
   }

/cc @crawshaw @rakyll

[rakyll]

There are two distinct problems in the scope of this issue. Although they are relatively connected, we need to address them individually.

Problems:

1. Name congestions among multiple packages.
2. Being able to name Java packages, using custom prefixes for headers on iOS.

The solution to the first problem is relatively easy. If there are name congestions, reject to generate bindings or name the packages.

An example:

 $ gomobile bind -target=android a/sensor b/sensor

It may generate two classes ASensor.java and BSensor.java. We can simply prefix the class name in the most minimal way it will be distinguishable from the other package. Given the fact that, Go import path is unique within the GOPATH, we will eventually find a unique name.

If the user is not happy with the auto-assigned names, they should be able to override the namespace themselves.

I don't have a strong preference how we should represent the mapping but the information shouldn't be sitting in a separate file. Would you like to carry the discussion to golang-dev@ to ask feedback from a wider audience?

[hyangah]

I will not worry too much about the collision in the package names at this moment. I assume the user has full control over what the package name will be. So, for two sensor packages, the user can name one in sensorA, and another in sensorB. For binding 3rd party packages, the user can wrap them with more meaningful names.

I think what I will end up implementing is

  $ gomobile bind -target=android -pkgpath=com.golang.app a.b.c x.y.z

This will creates two classes C.java and Z.java under the same package com.golang.app. All classes will be under the same package path (com.golang.app). If pkgpath is not given, go.c.C and go.z.Z.

For objective-c,

  $ gomobile bind -target=ios -prefix=App a.b.c x.y.z

This will create AppC.{h,m} and AppZ.{h.m}, and AppC and AppZ will be used as prefixes of generated functions and types. If -prefix is omitted, GoC and GoZ.

[crawshaw]

SGTM

[gopherbot]

CL https://golang.org/cl/13969 mentions this issue.

[sridharv]

For binding 3rd party packages, the user can wrap them with more meaningful names

Please reconsider this decision. One of the strengths with the Go ecosystem is that it is incredibly easy to share code. If I build a useful library and every developer will now have to manually write a wrapper the barrier to adoption is significantly higher. For example, I would love to just be able to go get a library to handle networking (checking network type WiFi/3G/etc) or location and just have it work. Having to now write a bunch of code to essentially re-implement the API imposes an unnecessary burden.

All of these problems are very easily solved if you just provide a separate command in gomobile to build. Then I can do the following easily:

$ gobind pkg1
$ gobind pkg2
$ gobind pkg3

Now manually create my main

package main

import (
    _ "pkg1"
    _ "pkg2"
    _ "pkg3"
)

func main() {
    panic("Does nothing!")
}

And build

$ gomobile build -target=ios -buildmode=objc-framework

This is consistent with how the go tool works and makes sharing code very easy. Even better, I can use go generate and it all fits beautifully into the existing go workflow. As an added bonus the source is now available for me to debug.

EDIT: I forgot to add, that the above approach has the advantage that it makes incremental builds possible. Having gobind being run automatically by Intellij/Android Studio/XCode is very helpful since it avoids build errors. Right now the framework / aar is rebuilt on every iteration.

[hyangah]

@sridharv Can you explain what the gobind does? Currently it does only code generation (.java, .h, .m, .go) but your mention of incremental builds makes me think the gobind in your proposal does something more than that.

• what's the output of the gobind? (source code? compiled object or class files? archive?)
• where should the gobind output go (so, the following gomobile build command finds the output of previous gobind commands and IDEs build task)? Every IDE has different assumptions on source code layout, or default build framework.

Use of buildmode flag in gomobile build is interesting, and I want to hear more about how it connects the main package and the product of previous gobind+IDE runs.

Besides, I want to know how to handle the case when exported functions of pkg1 and pkg2 both depend on some types of pkg3.

package pkg1
func Read(io.Reader)

package pkg2
func Work(io.Reader)

Our plan was to export only one type for io.Reader in this case, so both Read and Work use the same type.

I don't understand what prevents the use of go generate currently.

[sridharv]

Firstly, thank you for responding so quickly. Secondly, I should apologize for saying the problem is "very easily solved". It definitely comes off as glib, which I didn't realize when writing it. In addition I don't think I explained some of the usage scenarios I had in mind properly. Now onto your questions.

Besides, I want to know how to handle the case when exported functions of pkg1 and pkg2 both depend on some types of pkg3.

That's a very good point. What I described earlier doesn't address this properly since there is room for user error when running gobind manually for each package, which is never a good thing. Having said that, a single step for code generation makes the "separate bind and build" irrelevant for this specific issue. Now that I think about it, I was primarily motivated by some other problems I am currently facing with having the code gen and build be part of one step. I'll answer your questions regarding that in a separate comment and create a separate issue for the questions I have.

Proposal

There are two parts to this. One addresses the problem of manually wrapping libraries and the other, sharing cross platform code.

Binding multiple libraries in the face of package conflicts

Create a main file as you suggested in Option 3. Instead of adding comments per package just list all the packages. For packages that have name collisions use the standard Go mechanism of resolving name collisions.

package main

import (
    _ "pkg1"
    otherpkg1 "some/other/pkg1"
    _ "pkg2"
)

// Avoid the unused import warning
var _ = otherpkg1.SomeFunction

func main() {
    panic("placebo main")
}

Now run gomobile bind as you decided on earlier, but instead of listing all packages just point to this main package.

Java:

$ gomobile bind -target=android -pkgpath=com.golang.app path.to.main.package

Objective-C:

$ gomobile bind -target=ios -prefix=App path.to.main.package

This makes it consistent with how go resolves the same package namespace collision issue and keeps things simple as with your earlier decision. Things can be made even simpler if you disallow having any other exported members in the main package so you don't have to worry about how to bind those.

Sharing libraries with supporting Java or ObjC code

This is a separate issue, but it is affected by the choice of prefix/package path.

Consider the following scenario. I am writing an SDK app with go code shared between iOS and Android. I want to do the following in my shared go code:

• Know which directory to store my database files in.
• Know whether the network is on WiFi or 3G so I can decide whether or not to download a file.
• Run a download in the background.

All the above have standardized solutions in Android and iOS respectively. If I wanted a common interface to the above I could write a go library that accepts an interface and have Java/ObjC implement that interface. The go code then calls that interface to get the directory, network state, etc. This is the kind of code that can easily live in a third party library and make development easy for everyone. However, in order to do this, the library author needs to know what Java interface / ObjC protocol to implement in their Java / ObjC supporting code.

One solution is to allow package authors to specify their own Java/ObjCpackage prefix.

//Package network implements cross-platform support for network state on iOS and Android
package network // java:com.foo.network ios:GoNet

Packages that force a prefix are disallowed from having third party package references in their exported code. This ensures that the problem you described does not occur.

Now I can do the following:

$ go get gomobile/support # (or something similar)

And write go code like so:

import (
    "gomobile/support/network"
    "gomobile/support/storage"
)

func DoSomething() {
    switch network.Type() {
    case network.WiFi:
    // do something
    }
}

func DoSomethingElse() {
    // ... compute url
    location := filepath.Join(storage.UserDataDir(), "somefile")
    network.DownloadInBackground(url, location)
}

The network and storage packages would handle their platform specific code by using their SDK provided Java/ObjC code.

Distributing the supporting code

For this, one way would be to ask library authors to use the native code distribution options for their language. They would use Maven for Java and Cocoapods/Carthage for iOS. They just have to be careful not to package the generated binding code. Another would be allow them to place these files in a directory in their go tree. These files are then automatically copied by gomobile bind when it is generating the bindings for these packages. This makes things easier since a user of the library would just go get a library and be able to use it without having to worry about the right version of the supporting native code.

The gobind tool could follow the convention that any files found in a directory named java under a go package are copied to the generated java package. For ObjC a directory named objc is treated similarly. It would be simpler to have the Java and ObjC files live in the same directory as the go package, but this causes problems when building for android since the go tool complains about ObjC files being present without cgo being used.

I'll put together a small prototype describing the above scenarios so it is clearer.

[sridharv]

I'm placing my answers to your questions about code gen + build here.

what's the output of the gobind? (source code? compiled object or class files? archive?)

The outputs are still source files.

where should the gobind output go (so, the following gomobile build command finds the output of previous gobind commands and IDEs build task)? Every IDE has different assumptions on source code layout, or default build framework.

This is an important question and one I didn't address in my comment. I don't think IDEs need to automatically know about the location of the source. The only thing that needs to know is the gomobile tool so that it can pick up these files for the build step. This can be achieved with a command line flag.

Use of buildmode flag in gomobile build is interesting, and I want to hear more about how it connects the main package and the product of previous gobind+IDE runs

It can pick up the location of the files using a command line flag. The IDE will will pick the output of gomobile build in the same way it picks up the output of gomobile bind. The IDE can run gomobile build automatically (XCode Run Script Phase/Gradle plugin) whenever it builds the SDK app. However, the more I think about it this can be done in gomobile bind as well.

I don't understand what prevents the use of go generate currently.

Nothing whatsoever :). However, my understanding is that go generate is intended to be used for pre-build steps such as code generation. Library authors can then commit generated source to their library source tree. It feels like a misuse of go generate to use it to generate a .framework or .aar. Those operations feel like they should be done by go build/gomobile build/Makefile/build.gradle.

EDIT: I created #12819 to track my main concerns about code generation in gobind. Most of what I would like can be achieved without separating the two phases.

[sridharv]

I put together a prototype for reading the list of packages from a main file and specifying a package name override. It only works for gobind and is not fully tested right now. Depending on my time constraints I'll port it to gomobile and finish testing tomorrow.

sridharv/gomobile-java@710bb3c

[sridharv]

I currently need this feature for some work I'm doing. I'm happy to implement what was decided earlier on Aug 27 if you haven't had the time to look at it yet. Please let me know if this will be useful to you.

[sridharv]

I had time on Sunday and ended up implementing this (what was decided on Aug 27) for my personal use (using some of the earlier prototype code I'd written). The change is on the following branch: https://github.com/sridharv/mobile/tree/multibind2

The current status:

• Tested and working on Android.
• Referring to exported types from another package supported.
• Common generation code for gobind and gomobile bind.
• Custom prefixes/package paths not tested.
• Unit tests present for ObjC but not tested in an actual app.

If this will be useful I'll write up the approach I took, test for iOS and send the CL for review. I'm happy to make any changes needed. If it's not useful I'll use it for my work now and cut over to the official version when it lands :)