Go support: can't import generated protobuf file output

[pcj]

rules_protobuf go support is being developed on the "go" branch and originally based on @yugui https://github.com/grpc-ecosystem/grpc-gateway/blob/e958c5db30f7b99e1870db42dd5624322f112d0c/grpc-gateway.bzl.

Protocol buffer sources *.pb.go are successfully generated, but other go code that attempts to import the generated files fails:

$ git clone https://github.com/pubref/rules_protobuf
$ cd rules_protobuf
$ git checkout go
$ bazel build --verbose_failures examples/helloworld/go/client

WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:176:3: import_map: {"examples/helloworld/go/proto/helloworld.proto": "pubref.org/rules_pubref/examples/helloworld"}.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:256:5: protoc binary: bazel-out/host/genfiles/third_party/protoc/protoc.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:258:7:  > arg0: --plugin=protoc-gen-go=bazel-out/host/bin/external/com_github_golang_protobuf/protoc_gen_go.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:258:7:  > arg1: --go_out=Mexamples/helloworld/go/proto/helloworld.proto=pubref.org/rules_pubref/examples/helloworld,grpc:bazel-out/local-fastbuild/genfiles/.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:258:7:  > arg2: -I.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:258:7:  > arg3: examples/helloworld/go/proto/helloworld.proto.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:260:7:  > input: Artifact:[/Users/pcj/github/rules_protobuf[source]]examples/helloworld/go/proto/helloworld.proto.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:260:7:  > input: Artifact:[[/private/var/tmp/_bazel_pcj/63330772b4917b139280caef8bb81867/execroot/rules_protobuf]bazel-out/host/bin]external/com_github_golang_protobuf/protoc_gen_go.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:260:7:  > input: Artifact:[[/private/var/tmp/_bazel_pcj/63330772b4917b139280caef8bb81867/execroot/rules_protobuf]bazel-out/local-fastbuild/genfiles]examples/helloworld/go/proto/helloworld.proto.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:262:7:  > output: Artifact:[[/private/var/tmp/_bazel_pcj/63330772b4917b139280caef8bb81867/execroot/rules_protobuf]bazel-out/local-fastbuild/genfiles]examples/helloworld/go/proto/helloworld.pb.go.
WARNING: /Users/pcj/github/rules_protobuf/bzl/protoc.bzl:281:5: protoc_gen provided files: set([Artifact:[[/private/var/tmp/_bazel_pcj/63330772b4917b139280caef8bb81867/execroot/rules_protobuf]bazel-out/local-fastbuild/genfiles]examples/helloworld/go/proto/helloworld.pb.go]).
INFO: Found 1 target...
ERROR: /Users/pcj/github/rules_protobuf/examples/helloworld/go/client/BUILD:3:1: error executing shell command: 'rm -rf bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir && mkdir -p bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir && mkdir -p bazel-out/local-fast...' failed: bash failed: error executing command 
  (cd /private/var/tmp/_bazel_pcj/63330772b4917b139280caef8bb81867/execroot/rules_protobuf && \
  exec env - \
    GOARCH=amd64 \
    GOOS=darwin \
  /bin/bash -c '
rm -rf bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir 
&& mkdir -p bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir 
&& mkdir -p bazel-out/localfastbuild/bin/examples/helloworld/go/client/client.a.dir/github.com/golang 
&& ln -s ../../../../../../../../../../bazel-out/local-fastbuild/bin/external/com_github_golang_glog/go_default_library.a bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/github.com/golang/glog.a 
&& mkdir -p bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/github.com/golang/protobuf 
&& ln -s ../../../../../../../../../../../bazel-out/local-fastbuild/bin/external/com_github_golang_protobuf/proto.a bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/github.com/golang/protobuf/proto.a
&& mkdir -p bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/google.golang.org 
&& ln -s ../../../../../../../../../bazel-out/local-fastbuild/bin/external/org_golang_google_grpc/go_default_library.a bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/google.golang.org/grpc.a 
&& mkdir -p bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/golang.org/x/net 
&& ln -s ../../../../../../../../../../../bazel-out/local-fastbuild/bin/external/org_golang_x_net/context.a bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/golang.org/x/net/context.a 
&& mkdir -p bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/pubref.org/rules_protobuf/bazel-out/local-fastbuild/genfiles/examples/helloworld/go/proto 
&& ln -s ../../../../../../../../../../../../../../../../../bazel-out/local-fastbuild/genfiles/examples/helloworld/go/proto/helloworld.pb.go bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/pubref.org/rules_protobuf/bazel-out/local-fastbuild/genfiles/examples/helloworld/go/proto/helloworld.pb.go 
&& mkdir -p bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/pubref.org/rules_protobuf/examples/helloworld/go/client 
&& ln -s ../../../../../../../../../../../../../../examples/helloworld/go/client/main.go bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir/pubref.org/rules_protobuf/examples/helloworld/go/client/main.go 
&& export GOROOT=$(pwd)/external/golang_darwin_amd64/go/bin/.. 
&& cd  bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a.dir 
&& ../../../../../../../../external/golang_darwin_amd64/go/bin/go tool compile -o ../../../../../../../../bazel-out/local-fastbuild/bin/examples/helloworld/go/client/client.a -pack -I . pubref.org/rules_protobuf/examples/helloworld/go/client/main.go pubref.org/rules_protobuf/bazel-out/local-fastbuild/genfiles/examples/helloworld/go/proto/helloworld.pb.go'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
pubref.org/rules_protobuf/examples/helloworld/go/client/main.go:42: can't find import: "pubref.org/helloworld/examples/helloworld"
Target //examples/helloworld/go/client:client failed to build
INFO: Elapsed time: 0.225s, Critical Path: 0.10s

[pcj]

Looping in @hochhaus (bazelbuild/rules_closure#89) for possible review.

[hochhaus]

@pcj What are your goals for this project? Are you looking for full gRPC support or just basic go protobuf support for interop with rules_closure style JS?

If the latter, it might be easier to use a very light weight stub. For example, I'm using:

ahochhaus@ahochhaus-pc:~/ll$ cat go_proto_library.bzl 
load("@io_bazel_rules_go//go:def.bzl", "go_library")

def go_proto_library(
    name,
    srcs,
    visibility = None,
    testonly = 0,
    protoc = Label("@io_bazel_rules_closure//third_party/protobuf:protoc_bin"),
    protoc_gen_go = Label("@com_github_golang_protobuf//:protoc_gen_go")):
  cmd = [
    "$(location %s)" % protoc,
    "--plugin=protoc-gen-go=$(location %s)" % protoc_gen_go,
    "--go_out=$(GENDIR)",
  ]
  cmd += ["$(location " + src + ")" for src in srcs]

  go_srcs = [src.rsplit('.', 1)[0] + ".pb.go" for src in srcs]

  native.genrule(
      name = name + "_gen",
      srcs = srcs,
      testonly = testonly,
      visibility = visibility,
      message = "Generating Go Protocol Buffer file",
      outs = go_srcs,
      tools = [protoc, protoc_gen_go],
      cmd = " ".join(cmd),
  )

  go_library(
      name = name,
      srcs = go_srcs,
      deps = [
          str(Label("@com_github_golang_protobuf//:proto")),
      ],
  )

with a BUILD file that looks like

...

load("//:go_proto_library.bzl", "go_proto_library")

...

go_proto_library(
    name = "pb",
    srcs = ["pb/pb.proto"],
)

and then importing that package in go with:

import "<prefix>/pb"

(Obviously name it something better than "pb".)

I'm hesitant to put much time into a code review until we have a plan for where it can be upstreamed for general use. For example, if @yugui would consider accepting go_proto_library support in rules_go then I would happily send a change.

However, based upon bazelbuild/rules_closure#89 I think all of the bazel protobuf support will land in upstream protobuf eventually. So I'm not sure if it makes sense to store in rules_go for the time being.

What are your thoughts Yuki Yugui Sonoda?

[pcj]

Thanks @hochhaus. My goal is ultimately to provide comprehensive protobuf support for multiple languages, with gRPC support. Based on my review of sources in getting this started, it's a highly fragmented area given the diversity of tools and existing build infrastructures.

As for golang, I'm still getting my head around how package names, import statements, go_prefix, and go_library all interact. Specifically, the protoc-gen-go plugin has this concept of an ImportMap that is exposed via the option format --go_out=Mfoo=bar:dirname.

It's not too hard to generate the *.pb.go files, but have not been able to line it all up to get those files to compile in a different source file that imports them.

[pcj]

Andy @hochhaus if you send email to pcj@pubref.org I will cc: you on email thread with @yugui.

[pcj]

Note: golang support is now provided, but there are some limitations. See #4.