Golang git_repository and new_git_repository failure

[laramiel]

I have created a series of git repositories to demonstrate basic bazel usage:

https://github.com/laramiel/bazel-example-golang
https://github.com/laramiel/bazel-example-golang-remote
https://github.com/laramiel/bazel-example-golang-bare

I expect to be able to issue each of these commends and have it print out contents correctly.

bazel run :hello
bazel run :remote
bazel run :bare

However, instead I get this message:

ERROR: /Users/lar/github/bazel-example-golang/BUILD:25:1: error executing shell command: 'rm -rf bazel-out/local_darwin-fastbuild/bin/bare.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/bare.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/bare.a.dir/github.com/laramiel/ba...' failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
github.com/laramiel/bazel-example-golang/bare.go:5: can't find import: "github.com/laramiel/bazel-example-golang-bare/bare"
Target //:bare failed to build

Steps to repro:

1. Build bazel at head (to get the fix for the go language build rules)
2. Run bazel run :bare or bazel run :remote
3. Watch the failure.

[laramiel]

$ bazel run :remote --verbose_failures
bazel run :remote --verbose_failures
INFO: Found 1 target...
ERROR: /Users/lar/github/bazel-example-golang/BUILD:17:1: error executing shell command: 'rm -rf bazel-out/local_darwin-fastbuild/bin/remote.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/remote.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laram...' failed: bash failed: error executing command
  (cd /private/var/tmp/_bazel_lar/bb07b1ed47791f0f8c15114efe109bcd/bazel-example-golang && \
  exec env - \
    GOARCH=amd64 \
    GOOS=darwin \
  /bin/bash -c 'rm -rf bazel-out/local_darwin-fastbuild/bin/remote.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/remote.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang/external/remote && ln -s ../../../../../../../../../bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang/external/remote/remote.a && mkdir -p bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang && ln -s ../../../../../../../remote.go bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang/remote.go && export GOROOT=$(pwd)/external/golang_darwin_amd64/go/bin/.. && cd  bazel-out/local_darwin-fastbuild/bin/remote.a.dir && ../../../../external/golang_darwin_amd64/go/bin/go tool compile -o ../../../../bazel-out/local_darwin-fastbuild/bin/remote.a -pack -I . github.com/laramiel/bazel-example-golang/remote.go'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
github.com/laramiel/bazel-example-golang/remote.go:5: can't find import: "github.com/laramiel/bazel-example-golang-remote/remote"
Target //:remote failed to build

Breaking it down:

$ cd /private/var/tmp/_bazel_lar/bb07b1ed47791f0f8c15114efe109bcd/bazel-example-golang
$ find . | grep -v _bin
./.git
./.gitignore
./.gitmodules
./bare.go
./bazel-out
./bazel-out/local_darwin-fastbuild
./bazel-out/local_darwin-fastbuild/bin
./bazel-out/local_darwin-fastbuild/bin/external
./bazel-out/local_darwin-fastbuild/bin/external/remote
./bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a
./bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a.dir
./bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a.dir/github.com
./bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a.dir/github.com/laramiel
./bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a.dir/github.com/laramiel/bazel-example-golang-remote
./bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a.dir/github.com/laramiel/bazel-example-golang-remote/external
./bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a.dir/github.com/laramiel/bazel-example-golang-remote/external/remote
./bazel-out/local_darwin-fastbuild/bin/external/remote/remote.a.dir/github.com/laramiel/bazel-example-golang-remote/external/remote/remote.go
./bazel-out/local_darwin-fastbuild/bin/remote.a.dir
./bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com
./bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel
./bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang
./bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang/external
./bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang/external/remote
./bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang/external/remote/remote.a
./bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang/remote.go
./bazel-out/local_darwin-fastbuild/bin/remote.runfiles
./bazel-out/local_darwin-fastbuild/bin/remote.runfiles/MANIFEST
./bazel-out/local_darwin-fastbuild/bin/remote.runfiles/remote
./bazel-out/local_darwin-fastbuild/bin/remote.runfiles_manifest
./bazel-out/local_darwin-fastbuild/genfiles
./bazel-out/local_darwin-fastbuild/testlogs
./bazel-out/stable-status.txt
./bazel-out/volatile-status.txt
./BUILD
./external
./hello.go
./local
./README.md
./remote.go
./subbare.go
./submodule
./subremote.go
./tools
./WORKSPACE

Some of the symlinks setup links to the go compliler and root; ignoring those, the simplified command is:

$ cd  bazel-out/local_darwin-fastbuild/bin/remote.a.dir
$ go tool compile -o ../../../../bazel-out/local_darwin-fastbuild/bin/remote.a -pack -I . github.com/laramiel/bazel-example-golang/remote.go

github.com/laramiel/bazel-example-golang/remote.go:5: can't find import: "github.com/laramiel/bazel-example-golang-remote/remote"

[laramiel]

So the basic problem appears to be that the external repositories are not made available under the original name, and there is no rewrite / vendoring which corrects for this.

[laramiel]

Digging into the :remote case more.

The WORKSPACE file has this rule:

git_repository(
name = "remote",
remote = "https://github.com/laramiel/bazel-example-golang-remote.git",
commit = "8f2e405",
)

This will cause the referenced github repository to be downloaded into the bazel external folder under the name. Here, bazel-bazel-example-golang/external/remote.

As the build progresses--see above--the package file is eventually linked into the source tree. See the files list:

./bazel-out/local_darwin-fastbuild/bin/remote.a.dir/github.com/laramiel/bazel-example-golang/external/remote/remote.a

But the go too cannot find it because it has a new, and very different, name.
The golang command should be:

$ go tool compile -o ../../../../bazel-out/local_darwin-fastbuild/bin/remote.a -pack -I . -importmap github.com/laramiel/bazel-example-golang-remote/remote=github.com/laramiel/bazel-example-golang/external/remote/remote  github.com/laramiel/bazel-example-golang/remote.go

[laramiel]

So, potential problems that I see here.

1. emit_compile_action() does not take into account the go_prefix() value of dependencies.
   Libraries declared via the git_repository() / new_git_repository() mechanism will have different go_prefix() values, and will not work without writing an -importmap for them.
   I am not familiar enough with the issue to know how to access go_prefix() from a library object, but in order for this to work, the prefix value must be read from the library, compared with the prefix of the current rule, and, when different, cause the importmap to be generated correctly.

   So this seems to be a bug around build_rules/go/def.bzl:128 emit_go_compile_action()

2. A potential collision between git_repository() / new_git_repository() and a repository-level folder named external. I haven't investigate this too much, but given the sym-linking of external, it seems likely.

[hanwen]

The Go rules have no special provisions for working with remote repositories, so the support for that would have to be written.

Have you seen this thread, which deals with the same topic,

https://groups.google.com/forum/#!searchin/bazel-discuss/santiago/bazel-discuss/qeeQEQCQlo8/A3DYgQlwCgAJ

[laramiel]

I have updated the linked repositories so that they build, highlighting the workaround.

[dslomov]

@kchodorow any further action needed here?

[kchodorow]

I don't think so, @laramiel feel free to comment if you disagree.

[laramiel]

I disagree, unless there is already a tracking bug for working with external golang repositories.

[kchodorow]

We are moving the go rules (and all of the other rules) into rule-specific repositories, so I've opened a new umbrella issue in rules_go about it. Feel free to add any details there.