bazel build depends on *.pb.go files being generated first

[jml]

tl;dr:

• update-gazelle generates incorrect BUILD files unless *.pb.go files have been generated first
• Even with correct BUILD files, make bazel and make bazel-test will fail unless *.pb.go files have been generated first

Fix is to incorporate protobuf generation into BUILD files.

Complicated by the fact that we use a non-standard mechanism to generate our protobufs for undocumented reasons.

Investigation

Important: This starts from a clean checkout of cortex. Start with make clean to reproduce the issue.

I've been experimenting with bumping to latest bazel & rules_go.

$ bazel version
Build label: 0.5.3-homebrew
Build target: bazel-out/darwin_x86_64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jul 27 19:30:37 2017 (1501183837)
Build timestamp: 1501183837
Build timestamp as int: 1501183837

$ make bazel bazel-test
bazel build //cmd/...
ERROR: /private/var/tmp/_bazel_jml/c3efe7604fd21db2522d82194bf89683/external/io_bazel_rules_go/go/tools/extract_package/BUILD:4:1: in go_tool_binary rule @io_bazel_rules_go//go/tools/extract_package:extract_package:
Traceback (most recent call last):
	File "/private/var/tmp/_bazel_jml/c3efe7604fd21db2522d82194bf89683/external/io_bazel_rules_go/go/tools/extract_package/BUILD", line 4
		go_tool_binary(name = 'extract_package')
	File "/private/var/tmp/_bazel_jml/c3efe7604fd21db2522d82194bf89683/external/io_bazel_rules_go/go/private/go_tool_binary.bzl", line 35, in _go_tool_binary_impl
		ctx.action(inputs = ((ctx.files.srcs + tool...), <4 more arguments>)
	File "/private/var/tmp/_bazel_jml/c3efe7604fd21db2522d82194bf89683/external/io_bazel_rules_go/go/private/go_tool_binary.bzl", line 36, in ctx.action
		toolchain.tools
object of type 'Target' has no field 'tools'.
ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks-experiments/loki/pkg/client/BUILD.bazel:3:1: no such package 'vendor/github.com/openzipkin/zipkin-go-opentracing/_thrift/gen-go/zipkincore': BUILD file not found on package path and referenced by '//vendor/github.com/weaveworks-experiments/loki/pkg/client:go_default_library'.
ERROR: Analysis of target '//cmd/ingester:ingester' failed; build aborted.
INFO: Elapsed time: 0.202s
make: *** [bazel] Error 1

rules_go document that latest bazel requires rules_go 0.5.2 or better, so…

--- a/WORKSPACE
+++ b/WORKSPACE
@@ -1,7 +1,9 @@
 git_repository(
     name = "io_bazel_rules_go",
     remote = "https://github.com/bazelbuild/rules_go.git",
-    commit = "9bd1d8969862baac16540d44d2cf87f6a44e7734",
+    # Required for working with bazel 0.5.3
+    # https://github.com/bazelbuild/rules_go/tree/10a700af62d2c23cc6e9bbc12055c0abe2808a23
+    tag = "0.5.2",
 )
 load("@io_bazel_rules_go//go:def.bzl", "go_repositories")

This reduces the error to:

$ make bazel bazel-test
bazel build //cmd/...
ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks-experiments/loki/pkg/client/BUILD.bazel:3:1: no such package 'vendor/github.com/openzipkin/zipkin-go-opentracing/_thrift/gen-go/zipkincore': BUILD file not found on package path and referenced by '//vendor/github.com/weaveworks-experiments/loki/pkg/client:go_default_library'.
ERROR: Analysis of target '//cmd/table-manager:table-manager' failed; build aborted.
INFO: Elapsed time: 32.348s
make: *** [bazel] Error 1

Maybe I need to update the BUILD files for the new version of rules_go? I run make update-gazelle:

$ git st
M       vendor/github.com/Sirupsen/logrus/BUILD.bazel
M       vendor/github.com/hashicorp/go-rootcerts/BUILD.bazel
M       vendor/github.com/lib/pq/BUILD.bazel
M       vendor/github.com/mattn/go-colorable/BUILD.bazel
M       vendor/github.com/mattn/go-isatty/BUILD.bazel
M       vendor/github.com/miekg/dns/BUILD.bazel
M       vendor/github.com/prometheus/common/log/BUILD.bazel
M       vendor/github.com/prometheus/prometheus/util/flock/BUILD.bazel
M       vendor/github.com/syndtr/goleveldb/leveldb/storage/BUILD.bazel
M       vendor/github.com/tinylib/msgp/msgp/BUILD.bazel
M       vendor/golang.org/x/sys/unix/BUILD.bazel
M       vendor/golang.org/x/sys/windows/BUILD.bazel
M       vendor/golang.org/x/sys/windows/registry/BUILD.bazel
M       vendor/golang.org/x/sys/windows/svc/BUILD.bazel
M       vendor/golang.org/x/sys/windows/svc/eventlog/BUILD.bazel
M       vendor/gopkg.in/fsnotify.v1/BUILD.bazel
M       vendor/k8s.io/client-go/1.5/pkg/util/BUILD.bazel

The same problem (i.e. something wrong with zipkin-go-opentracing) occurs when I run make bazel.

I don't really know what's going on here, but if I hit zipkin-go-opentracing with the upgrade hammer…

diff --git a/Gopkg.toml b/Gopkg.toml
index 63e4fa1..5595c1a 100644
--- a/Gopkg.toml
+++ b/Gopkg.toml
@@ -18,3 +18,10 @@
 [[override]]
   name = "github.com/weaveworks/mesh"
   revision = "5015f896ab62d3e9fe757456c757521ce0c3faff"
+
+# bazel build does not work with earlier versions:
+#
+# ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks-experiments/loki/pkg/client/BUILD.bazel:3:1: no such package 'vendor/github.com/openzipkin/zipkin-go-opentracing/_thrift/gen-go/zipkincore': BUILD file not found on package path and referenced by '//vendor/github.com/weaveworks-experiments/loki/pkg/client:go_default_library'.
+[[override]]
+  name = "github.com/openzipkin/zipkin-go-opentracing"
+  version = "0.3.0"

$ make update-vendor
dep ensure && dep prune
git status | grep BUILD.bazel | cut -d' ' -f 5 | xargs git checkout HEAD
$ make update-gazelle
gazelle -go_prefix github.com/weaveworks/cortex -external vendored \
		-build_file_name BUILD.bazel

The problem becomes:

$ make bazel bazel-test
bazel build //cmd/...
INFO: Found 16 targets...
ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/pkg/ingester/client/BUILD.bazel:3:1: GoCompile pkg/ingester/client/go_default_library.o failed (Exit 1).
pkg/ingester/client/client.go:15: undefined: IngesterClient
2017/08/05 11:36:58 error running compiler: exit status 1
INFO: Elapsed time: 0.822s, Critical Path: 0.11s
make: *** [bazel] Error 1

Looking at the BUILD file in question:

filegroup(
    name = "go_default_library_protos",
    srcs = ["cortex.proto"],
    visibility = ["//visibility:public"],
)

That seems a weird way to build protobufs to me, in that it won't actually generate any Go (i.e. *.pb.go) files.

I tried using the documented way of generating protobuf libraries…

load("@io_bazel_rules_go//proto:go_proto_library.bzl", "go_proto_library")

go_proto_library(
    name = "go_default_library_protos",
    srcs = ["cortex.proto"],
    visibility = ["//visibility:public"],
)

But that errors with:

$ make bazel bazel-test
bazel build //cmd/...
ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/pkg/ingester/client/BUILD.bazel:22:1: no such package '@com_github_golang_protobuf//protoc-gen-go': The repository could not be resolved and referenced by '//pkg/ingester/client:go_default_library_protos_protos'.

(The duplicated _protos suffix leads me to believe I am misusing go_proto_library somehow).

At this point, I recall that we use a non-standard tool (https://github.com/gogo/protobuf) for generating our Go protobuf libraries (although I can't recall why). I would like to understand this better, but for now I'm setting it aside.

My poorly-formed hypothesis is that the bazel BUILD files were somehow assuming that *.pb.go exists and were implicitly making them available to the Go package and that something in rules_go or bazel has changed to make this not work.

To experiment, I run make to generate the *.pb.go files and then run make update-gazelle. This indeed updates the relevant BUILD files:

diff --git a/pkg/ingester/client/BUILD.bazel b/pkg/ingester/client/BUILD.bazel
index ae78bab..014c73a 100644
--- a/pkg/ingester/client/BUILD.bazel
+++ b/pkg/ingester/client/BUILD.bazel
@@ -4,15 +4,19 @@ go_library(
     name = "go_default_library",
     srcs = [
         "client.go",
+        "cortex.pb.go",
         "dep.go",
     ],
     visibility = ["//visibility:public"],
     deps = [
+        "//pkg/util/wire:go_default_library",
         "//vendor/github.com/gogo/protobuf/gogoproto:go_default_library",
+        "//vendor/github.com/gogo/protobuf/proto:go_default_library",
         "//vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc:go_default_library",
         "//vendor/github.com/mwitkow/go-grpc-middleware:go_default_library",
         "//vendor/github.com/opentracing/opentracing-go:go_default_library",
         "//vendor/github.com/weaveworks/common/middleware:go_default_library",
+        "//vendor/golang.org/x/net/context:go_default_library",
         "//vendor/google.golang.org/grpc:go_default_library",
     ],
 )
diff --git a/pkg/ring/BUILD.bazel b/pkg/ring/BUILD.bazel
index a8af61c..35d1709 100644
--- a/pkg/ring/BUILD.bazel
+++ b/pkg/ring/BUILD.bazel
@@ -8,11 +8,14 @@ go_library(
         "http.go",
         "model.go",
         "ring.go",
+        "ring.pb.go",
         "util.go",
     ],
     visibility = ["//visibility:public"],
     deps = [
         "//pkg/util:go_default_library",
+        "//vendor/github.com/gogo/protobuf/proto:go_default_library",
+        "//vendor/github.com/gogo/protobuf/sortkeys:go_default_library",
         "//vendor/github.com/golang/protobuf/proto:go_default_library",
         "//vendor/github.com/golang/snappy:go_default_library",
         "//vendor/github.com/hashicorp/consul/api:go_default_library",

And make bazel bazel-test now works!

If I remove the generated proto files and run bazel again, it fails, but with a much clearer error message:

$ find pkg -name '*.pb.go' | xargs rm
jml@gallantry:~/go/src/github.com/weaveworks/cortex (newer-bazel) (0)
$ make bazel bazel-test
bazel build //cmd/...
INFO: Found 16 targets...
ERROR: missing input file '//pkg/ring:ring.pb.go'.
ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/pkg/ring/BUILD.bazel:3:1: //pkg/ring:go_default_library: missing input file '//pkg/ring:ring.pb.go'.
ERROR: /Users/jml/go/src/github.com/weaveworks/cortex/pkg/ring/BUILD.bazel:3:1 1 input file(s) do not exist.
INFO: Elapsed time: 0.187s, Critical Path: 0.01s
make: *** [bazel] Error 1

[jml]

Hit Submit too early. Still editing main comment.

[jml]

I just saw the zipkin-go-opentracing thing has a bug, bazel-contrib/rules_go#423, which has since been fixed. I wonder if this means we can stop committing the generated BUILD files, and just have bazel use gazelle directly? @tomwilkie WDYT?

[tomwilkie]

Complicated by the fact that we use a non-standard mechanism to generate our protobufs for undocumented reasons.

I documented the reasons extensively, and publicly: https://www.weave.works/blog/the-long-tail-tools-to-investigate-high-long-tail-latency/

I wonder if this means we can stop committing the generated BUILD files, and just have bazel use gazelle directly? @tomwilkie WDYT?

Personally I'd leave them committed, as your going to have to add special rules for the protobufs anyway. Eventually the aim is to remove the old makefile-based builds, so the BUILD files getting out of sync won't be an issue.

[jml]

I documented the reasons extensively, and publicly: https://www.weave.works/blog/the-long-tail-tools-to-investigate-high-long-tail-latency/

Thanks. I didn't think to look there.

[tomwilkie]

As discussed on the community call, we're dropping bazel as an way of building cortex.