Bazel skylark rules for building protocol buffers with +/- gRPC support on (osx, linux) ✨.
 |
 |
 |
| Bazel | rules_protobuf | gRPC |
These rules sprung out of a need to have protobuf support when there was limited exposed and documented proto generation capabilities in the main bazel repository. This is a moving target. The main goals of this project are to:
-
Provide
protoc, the protocol buffer compiler (v3.1.x52ab). -
Provide the language-specific plugins.
-
Provide the necessary libraries and dependencies for gRPC support, when possible.
-
Provide an extensible
proto_languageabstraction (used in conjunction with theproto_compilerule) to generate outputs for current and future custom protoc plugins not explicitly provided here.
-
Support for generation of protoc outputs via
proto_compile()rule. -
Support for generation + compilation of outputs with protobuf dependencies.
-
gRPC support.
-
Highly experimental (probably not functional yet). A work-in-progress for those interested in contributing further work.
These are build rules for bazel. If you have not already
installed bazel on your workstation, follow the
bazel instructions.
Bazel 0.3.1 or above is required for go support.
Note about protoc and related tools: bazel and rules_protobuf will download or build-from-source all required dependencies, including the
protoctool and required plugins. If you do already have these tools installed on your workstation, bazel will not use them.
Specify the language(s) you'd like use by loading the
language-specific *_proto_repositories rule(s):
git_repository(
name = "org_pubref_rules_protobuf",
remote = "https://github.com/pubref/rules_protobuf",
tag = "v0.7.1",
#commit = "..." # alternatively, latest commit on master
)
load("@org_pubref_rules_protobuf//java:rules.bzl", "java_proto_repositories")
java_proto_repositories()
load("@org_pubref_rules_protobuf//cpp:rules.bzl", "cpp_proto_repositories")
cpp_proto_repositories()
load("@org_pubref_rules_protobuf//go:rules.bzl", "go_proto_repositories")
go_proto_repositories()Several languages have other rules_* dependencies that you'll need
to load before the *_proto_repositories() function is invoked:
| Language | Requires |
|---|---|
| go_proto_repositories | rules_go |
| gogo_proto_repositories | rules_go |
| grpc_gateway_proto_repositories | rules_go |
| closure_proto_repositories | rules_closure |
| csharp_proto_repositories | rules_dotnet |
| node_proto_repositories | rules_node |
If you're only interested in the proto_compile rule and not any
language-specific rules, just load the generic proto_repositories
rule. This provides the minimal set of dependencies (only the
protoc tool).
load("@org_pubref_rules_protobuf//protobuf:rules.bzl", "proto_repositories")
proto_repositories()To build a java-based gRPC library:
load("@org_pubref_rules_protobuf//java:rules.bzl", "java_proto_library")
java_proto_library(
name = "protolib",
protos = [
"my.proto"
],
with_grpc = True,
verbose = 1, # 0=no output, 1=show protoc command, 2+ more...
)To run the examples & tests in this repository, clone it to your workstation.
# Clone this repo
$ git clone https://github.com/pubref/rules_protobuf
# Go to examples/helloworld directory
$ cd rules_protobuf/examples/helloworld
# Run all tests
$ bazel test examples/...
# Build a server
$ bazel build cpp/server
# Run a server from the command-line
$ $(bazel info bazel-bin)/examples/helloworld/cpp/server
# Run a client
$ bazel run go/client
$ bazel run cpp/client
$ bazel run java/org/pubref/rules_closure/examples/helloworld/client:nettyTo load alternate versions of dependencies, pass in a dict having the same overall structure of a [deps.bzl][protobuf/deps.bzl] file. Entries having a matching key will override those found in the file. For example, to load a different version of https://github.com/golang/protobuf, provide a different commit ID:
load("@org_pubref_rules_protobuf//go:rules.bzl", "go_proto_repositories")
go_proto_repositories(
overrides = {
"com_github_golang_protobuf": {
"commit": "2c1988e8c18d14b142c0b472624f71647cf39adb", # Aug 8, 2016
}
},
)You may already have some external dependencies already present in
your workspace that rules_protobuf will attempt to load, causing a
collision. To prevent rules_protobuf from loading specific external
workspaces, name them in the excludes list:
go_proto_repositories(
excludes = [
"com_github_golang_glog",
]
)To completely replace the set of dependencies that will attempt to be
loaded, you can pass in a full dict object to the lang_deps
attribute.
go_proto_repositories(
lang_deps = {
"com_github_golang_glog": {
...
},
},
)There are several language --> language dependencies as well. For
example, python_proto_repositories and ruby_proto_repositories
(and more) internally call the cpp_proto_repositories rule to
provide the grpc plugins. To suppress this (and have better control
in your workspace), you can use the omit_cpp_repositories=True
option.
Use the proto_deps attribute to name proto rule dependencies. Use of
proto_deps implies you're using imports, so read on...
In all cases, these rules will include a --proto_path=. (-I.)
argument. This is functionally equivalent to --proto_path=$(bazel info execution_root). Therefore, when the protoc tool is invoked, it
will 'see' whatever directory structure exists at the bazel execution
root for your workspace. To better learn what this looks like, cd $(bazel info execution_root) and look around. In general, it
contains all your sourcefiles as they appear in your workspace with an
additional external/WORKSPACE_NAME directory for all dependencies
used.
This has implications for import statements in your protobuf sourcefiles, if you use them. The two cases to consider are imports within your workspace (referred to here as 'internal' imports), and imports of other protobuf files in an external workspace (external imports).
Internal imports should require no additional parameters if your
import statements follow the same directory structure of your
workspace. For example, the
examples/helloworld/proto/helloworld.proto file imports the
examples/proto/common.proto file. Since this matches the workspace
directory structure, protoc can find it, and no additional arguments
to a cc_proto_library are required for protoc code generation step.
Obviously, importing a file does not mean that code will be generated
for it. Therefore, use of the imports attribute implies that the
generated files for the imported message or service already exist
somewhere that can be used as a dependency some other library rule
(such as srcs for java_library).
Rather than using imports, it often make more sense to declare a
dependency on another proto_library rule via the proto_deps
attribute. This makes the import available to the calling rule and
performs the code generation step. For example, the
cc_proto_library rule in examples/helloworld/proto:cpp names the
//examples/proto:cpp's cc_proto_library rule in its proto_deps
attribute to accomplish both code generation and compilation of object
files for the proto chain.
The same logic applies to external imports. The two questions to ask yourself when setting up your rules are:
[Question 1]: Can protoc "see" the imported file? In order to satisfy this
requirement, pass in the full path of the required file(s) relative to
the execution root where protoc will be run. For example, the the
well-known descriptor.proto could be made visible to protoc via:
java_proto_library(
name = 'fooprotos',
protos = 'foo.proto`,
imports = [
"external/com_github_google_protobuf/src/",
],
inputs = [
"@com_github_google_protobuf//:well_known_protos",
],
)This would be imported as import "google/protobuf/descriptor.proto"
given that the file
@com_github_google_protobuf/src/google/protobuf/descriptor.proto is
in the package google.protobuf.
[Question 2]: Can the cc_proto_library rule "see" the generated protobuf files?
(in this case descriptor.pb.{h,cc}. Just because the file was
imported does not imply that protoc will generate outputs for it, so
somewhere in the cc_library rule dependency chain these files must
be present. This could be via another cc_proto_library rule defined
elswhere, or a some other filegroup or label list. If the source is
another cc_proto_library rule, specify that in the proto_deps
attribute to the calling cc_proto_library rule. Otherwise, pass a
label that includes the (pregenerated) protobuf files to the deps
attribute, just as you would any typical cc_library rule.
Important note about sandboxing: simply stating the path where
protoc should look for imports (via the imports attribute) is not
enough to work with the bazel sandbox. Bazel is very particular
about needing to know exactly which inputs are required for a
rule, and exactly what output files it generates. If an input is
not declared, it will not be exposed in the sandbox. Therefore, we
have to provide both the import path and a label-generating rule
in the inputs attribute that names the files we want available in
the sandbox (given here by :well_known_protos).
If you are having problems, put verbose={1,2,3} in your build rule
and/or disable sandboxing with --spawn_strategy=standalone.
Note: bazel will likely a breaking change to the way external repositories are laid out in the execution root in future versions of bazel. This will likely affect the naming of import paths when this change occurs.
Contributions welcome; please create Issues or GitHub pull requests.
-
@yugui: Primary source for the go support from grpc-gateway.
-
@mzhaom: Primary source for the skylark rule (from https://github.com/mzhaom/trunk/blob/master/third_party/grpc/grpc_proto.bzl).
-
@jart: Overall repository structure and bazel code layout (based on rules_closure).
-
@korfuri: Prior research on travis-ci integration.
-
Much thanks to all the members of the bazel, protobuf, and gRPC teams.