Skip to content
Modern bazel build rules for protobuf / gRPC
Branch: master
Clone or download
pcj Merge pull request #104 from Globegitter/update-rules-go
Update rules_go for bazel 0.27 compatibility.
Latest commit d9a1230 Jun 27, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.bazelci Use all_incompatible_changes by default (#75) May 7, 2019
android
closure grpc.js: split generated js compilation (#81) May 8, 2019
cpp Enable creation of gRPC mock code for C++. (#51) May 7, 2019
csharp Use all_incompatible_changes by default (#75) May 7, 2019
d Use all_incompatible_changes by default (#75) May 7, 2019
dart Use all_incompatible_changes by default (#75) May 7, 2019
example Be compatible with incompatible_restrict_named_params (#96) Jun 4, 2019
github.com grpc.js: use updated suppression list (#82) May 8, 2019
go Use all_incompatible_changes by default (#75) May 7, 2019
java java/scala: remove usage of maven_jar. (#90) May 10, 2019
node Use all_incompatible_changes by default (#75) May 7, 2019
objc Use all_incompatible_changes by default (#75) May 7, 2019
php Use all_incompatible_changes by default (#75) May 7, 2019
protobuf Update for bazel 0.24.1 (#69) May 4, 2019
python
ruby Use all_incompatible_changes by default (#75) May 7, 2019
rust Fix ci badge image link (#76) May 7, 2019
scala Improve documentation of generated files (#97) Jun 4, 2019
swift Use all_incompatible_changes by default (#75) May 7, 2019
tools Improve documentation of generated files (#97) Jun 4, 2019
.bazelrc turn off incompatible_tls_enabled_removed Jun 26, 2019
.gitignore Update for bazel 0.24.1 (#69) May 4, 2019
.pre-commit-config.yaml feat: add linter script that purifies text files Dec 18, 2018
BUILD.bazel Update for bazel 0.24.1 (#69) May 4, 2019
LICENSE Add android rules Sep 28, 2018
Makefile Improve documentation of generated files (#97) Jun 4, 2019
README.md Improve documentation of generated files (#97) Jun 4, 2019
WORKSPACE Fix ci badge image link (#76) May 7, 2019
aspect.bzl python: gather transitive outputs from the aspect graph. (#89) May 9, 2019
compile.bzl Remove format specifier from verbose output logs (#100) Jun 21, 2019
deps.bzl Update rules_go for bazel 0.27 compatibility. Jun 26, 2019
plugin.bzl Apply buildifier Dec 23, 2018

README.md

rules_proto

Bazel skylark rules for building protocol buffers +/- gRPC .

Bazel rules_proto gRPC

Rules

Status Lang Rule Description
Build Status android android_proto_compile Generates android protobuf artifacts (example)
Build Status android android_grpc_compile Generates android protobuf+gRPC artifacts (example)
Build Status android android_proto_library Generates android protobuf library (example)
Build Status android android_grpc_library Generates android protobuf+gRPC library (example)
Build Status closure closure_proto_compile Generates closure *.js protobuf+gRPC files (example)
Build Status closure closure_proto_library Generates a closure_library with compiled protobuf *.js files (example)
Build Status cpp cpp_proto_compile Generates .h,.cc protobuf artifacts (example)
Build Status cpp cpp_grpc_compile Generates .h,.cc protobuf+gRPC artifacts (example)
Build Status cpp cpp_proto_library Generates .h,.cc protobuf library (example)
Build Status cpp cpp_grpc_library Generates .h,.cc protobuf+gRPC library (example)
Build Status csharp csharp_proto_compile Generates csharp protobuf artifacts (example)
Build Status csharp csharp_grpc_compile Generates csharp protobuf+gRPC artifacts (example)
Build Status csharp csharp_proto_library Generates csharp protobuf library (example)
Build Status csharp csharp_grpc_library Generates csharp protobuf+gRPC library (example)
Build Status d d_proto_compile Generates d protobuf artifacts (example)
Build Status dart dart_proto_compile Generates dart protobuf artifacts (example)
Build Status dart dart_grpc_compile Generates dart protobuf+gRPC artifacts (example)
Build Status dart dart_proto_library Generates dart protobuf library (example)
Build Status dart dart_grpc_library Generates dart protobuf+gRPC library (example)
Build Status go go_proto_compile Generates *.go protobuf artifacts (example)
Build Status go go_grpc_compile Generates *.go protobuf+gRPC artifacts (example)
Build Status go go_proto_library Generates *.go protobuf library (example)
Build Status go go_grpc_library Generates *.go protobuf+gRPC library (example)
Build Status java java_proto_compile Generates a srcjar with protobuf *.java files (example)
Build Status java java_grpc_compile Generates a srcjar with protobuf+gRPC *.java files (example)
Build Status java java_proto_library Generates a jar with compiled protobuf *.class files (example)
Build Status java java_grpc_library Generates a jar with compiled protobuf+gRPC *.class files (example)
Build Status node node_proto_compile Generates node *.js protobuf artifacts (example)
Build Status node node_grpc_compile Generates node *.js protobuf+gRPC artifacts (example)
Build Status node node_proto_library Generates node *.js protobuf library (example)
Build Status node node_grpc_library Generates node *.js protobuf+gRPC library (example)
Build Status objc objc_proto_compile Generates objc protobuf artifacts (example)
Build Status objc objc_grpc_compile Generates objc protobuf+gRPC artifacts (example)
Build Status php php_proto_compile Generates php protobuf artifacts (example)
Build Status php php_grpc_compile Generates php protobuf+gRPC artifacts (example)
Build Status python python_proto_compile Generates *.py protobuf artifacts (example)
Build Status python python_grpc_compile Generates *.py protobuf+gRPC artifacts (example)
Build Status python python_proto_library Generates *.py protobuf library (example)
Build Status python python_grpc_library Generates *.py protobuf+gRPC library (example)
Build Status ruby ruby_proto_compile Generates *.ruby protobuf artifacts (example)
Build Status ruby ruby_grpc_compile Generates *.ruby protobuf+gRPC artifacts (example)
Build Status ruby ruby_proto_library Generates *.rb protobuf library (example)
Build Status ruby ruby_grpc_library Generates *.rb protobuf+gRPC library (example)
Build Status rust rust_proto_compile Generates rust protobuf artifacts (example)
experimental rust rust_grpc_compile Generates rust protobuf+gRPC artifacts (example)
Build Status rust rust_proto_library Generates rust protobuf library (example)
experimental rust rust_grpc_library Generates rust protobuf+gRPC library (example)
Build Status scala scala_proto_compile Generates *.scala protobuf artifacts (example)
Build Status scala scala_grpc_compile Generates *.scala protobuf+gRPC artifacts (example)
Build Status scala scala_proto_library Generates *.scala protobuf library (example)
Build Status scala scala_grpc_library Generates *.scala protobuf+gRPC library (example)
experimental swift swift_proto_compile Generates swift protobuf artifacts (example)
experimental swift swift_grpc_compile Generates swift protobuf+gRPC artifacts (example)
experimental swift swift_proto_library Generates swift protobuf library (example)
experimental swift swift_grpc_library Generates swift protobuf+gRPC library (example)
Build Status gogo gogo_proto_compile Generates gogo protobuf artifacts (example)
Build Status gogo gogo_grpc_compile Generates gogo protobuf+gRPC artifacts (example)
Build Status gogo gogo_proto_library Generates gogo protobuf library (example)
Build Status gogo gogo_grpc_library Generates gogo protobuf+gRPC library (example)
Build Status gogo gogofast_proto_compile Generates gogofast protobuf artifacts (example)
Build Status gogo gogofast_grpc_compile Generates gogofast protobuf+gRPC artifacts (example)
Build Status gogo gogofast_proto_library Generates gogofast protobuf library (example)
Build Status gogo gogofast_grpc_library Generates gogofast protobuf+gRPC library (example)
Build Status gogo gogofaster_proto_compile Generates gogofaster protobuf artifacts (example)
Build Status gogo gogofaster_grpc_compile Generates gogofaster protobuf+gRPC artifacts (example)
Build Status gogo gogofaster_proto_library Generates gogofaster protobuf library (example)
Build Status gogo gogofaster_grpc_library Generates gogofaster protobuf+gRPC library (example)
Build Status grpc-gateway gateway_grpc_compile Generates grpc-gateway *.go files (example)
Build Status grpc-gateway gateway_swagger_compile Generates grpc-gateway swagger *.json files (example)
Build Status grpc-gateway gateway_grpc_library Generates grpc-gateway library files (example)
Build Status grpc.js closure_grpc_compile Generates protobuf closure grpc *.js files (example)
Build Status grpc.js closure_grpc_library Generates protobuf closure library *.js files (example)
Build Status grpc-web closure_grpc_compile Generates a closure *.js protobuf+gRPC files (example)
Build Status grpc-web commonjs_grpc_compile Generates a commonjs *.js protobuf+gRPC files (example)
Build Status grpc-web commonjs_dts_grpc_compile Generates a commonjs_dts *.js protobuf+gRPC files (example)
Build Status grpc-web ts_grpc_compile Generates a commonjs *.ts protobuf+gRPC files (example)
Build Status grpc-web closure_grpc_library Generates protobuf closure library *.js files (example)

Overview

These rules are the successor to https://github.com/pubref/rules_protobuf and are in a pre-release status. The primary goals are:

  1. Interoperate with the native proto_library rules and other proto support in the bazel ecosystem as much as possible.
  2. Provide a proto_plugin rule to support custom protoc plugins.
  3. Minimal dependency loading. Proto rules should not pull in more dependencies than they absolutely need.

NOTE: in general, try to use the native proto library rules when possible to minimize external dependencies in your project. Add rules_proto when you have more complex proto requirements such as when dealing with multiple output languages, gRPC, unsupported (native) language support, or custom proto plugins.

Installation

Add rules_proto your WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "build_stack_rules_proto",
    urls = ["https://github.com/stackb/rules_proto/archive/b93b544f851fdcd3fc5c3d47aee3b7ca158a8841.tar.gz"],
    sha256 = "c62f0b442e82a6152fcd5b1c0b7c4028233a9e314078952b6b04253421d56d61",
    strip_prefix = "rules_proto-b93b544f851fdcd3fc5c3d47aee3b7ca158a8841",
)

Important: Follow instructions in the language-specific README.md for additional workspace dependencies that may be required.

Usage

Step 1: write a protocol buffer file (example: thing.proto):

syntax = "proto3";

package example;

import "google/protobuf/any.proto";

message Thing {
    string name = 1;
    google.protobuf.Any payload = 2;
}

Step 2: write a BAZEL.build file with a native proto_library rule:

proto_library(
    name = "thing_proto",
    srcs = ["thing.proto"],
    deps = ["@com_google_protobuf//:any_proto"],
)

In this example we have a dependency on a well-known type any.proto hance the proto_library to proto_library dependency.

Step 3: add a cpp_proto_compile rule (substitute cpp_ for the language of your choice).

NOTE: In this example thing.proto does not include service definitions (gRPC). For protos with services, use the cpp_grpc_compile rule instead.

# BUILD.bazel
load("@build_stack_rules_proto//cpp:cpp_proto_compile.bzl", "cpp_proto_compile")

cpp_proto_compile(
    name = "cpp_thing_proto",
    deps = [":thing_proto"],
)

But wait, before we can build this, we need to load the dependencies necessary for this rule (from cpp/README.md):

Step 4: load the workspace macro corresponding to the build rule.

# WORKSPACE
load("@build_stack_rules_proto//cpp:deps.bzl", "cpp_proto_compile")

cpp_proto_compile()

Note that the workspace macro has the same name of the build rule. Refer to the cpp/deps.bzl for details on what other dependencies are loaded.

We're now ready to build the rule:

Step 5: build it.

$ bazel build //example/proto:cpp_thing_proto
Target //example/proto:cpp_thing_proto up-to-date:
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.h
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.cc

If we were only interested for the generated file artifacts, the cpp_grpc_compile rule would be fine. However, for convenience we'd rather have the outputs compiled into an *.so file. To do that, let's change the rule from cpp_proto_compile to cpp_proto_library:

# WORKSPACE
load("@build_stack_rules_proto//cpp:deps.bzl", "cpp_proto_library")

cpp_proto_library()
# BUILD.bazel
load("@build_stack_rules_proto//cpp:cpp_proto_library.bzl", "cpp_proto_library")

cpp_proto_library(
    name = "cpp_thing_proto",
    deps = [":thing_proto"],
)
$ bazel build //example/proto:cpp_thing_proto
Target //example/proto:cpp_thing_proto up-to-date:
  bazel-bin/example/proto/libcpp_thing_proto.a
  bazel-bin/example/proto/libcpp_thing_proto.so  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.h
  bazel-genfiles/example/proto/cpp_thing_proto/example/proto/thing.pb.cc

This way, we can use //example/proto:cpp_thing_proto as a dependency of any other cc_library or cc_binary rule as per normal.

NOTE: the cpp_proto_library implicitly calls cpp_proto_compile, and we can access that rule by adding _pb at the end of the rule name, like bazel build //example/proto:cpp_thing_proto_pb.

Summary

  • There are generally four rule flavors for any language {lang}: {lang}_proto_compile, {lang}_proto_library, {lang}_grpc_compile, and {lang}_grpc_library.

  • If you are solely interested in the source code artifacts, use the {lang}_{proto|grpc}_compile rule.

  • If your proto file has services, use the {lang}_grpc_{compile|library} rule instead.

  • Load any external dependencies needed for the rule via the load("@build_stack_rules_proto//{lang}:deps.bzl", "{lang}_{proto|grpc}_{compile|library}").

Developers

Code Layout

Each language {lang} has a top-level subdirectory that contains:

  1. {lang}/README.md: generated documentation for the rule(s).

  2. {lang}/deps.bzl: contains macro functions that declare repository rule dependencies for that language. The name of the macro corresponds to the name of the build rule you'd like to use.

  3. {lang}/{rule}.bzl: rule implementation(s) of the form {lang}_{kind}_{type}, where kind is one of proto|grpc and type is one of compile|library.

  4. {lang}/BUILD.bazel: contains proto_plugin() declarations for the available plugins for that language.

  5. {lang}/example/{rule}/: contains a generated WORKSPACE and BUILD.bazel demonstrating standalone usage.

  6. {lang}/example/routeguide/: routeguide example implementation, if possible.

The root directory contains the base rule defintions:

  • plugin.bzl: A build rule that defines the name, tool binary, and options for a particular proto plugin.

  • compile.bzl: A build rule that contains the proto_compile rule. This rule calls protoc with a given list of plugins and generates output files.

Additional protoc plugins and their rules are scoped to the github repository name where the plugin resides. For example, there are 3 grpc-web implementations in this repo: `github.com/improbable-eng/grpc-web, github.com/grpc/grpc-web, and github.com/stackb/grpc.js.

Rule Generation

To help maintain consistency of the rule implementations and documentation, all of the rule implementations are generated by the tool //tools/rulegen. Changes in the main README.md should be placed in tools/rulegen/README.header.md or tools/rulegen/README.footer.md. Changes to generated rules should be put in the source files (example: tools/rulegen/java.go).

Transitivity

Briefly, here's how the rules work:

  1. Using the proto_library graph, collect all the *.proto files directly and transitively required for the protobuf compilation.

  2. Copy the *.proto files into a "staging" area in the bazel sandbox such that a single -I. will satisfy all imports.

  3. Call protoc OPTIONS FILELIST and generate outputs.

The concept of transitivity (as defined here) affects which files in the set *.proto files are named in the FILELIST. If we only list direct dependencies then we say transitive = False. If all files are named, then transitive = True. The set of files that can be included or excluded from the FILELIST are called transitivity rules, which can be defined on a per-rule or per-plugin basis. Please grep the codebase for examples of their usage.

Developing Custom Plugins

Follow the pattern seen in the multiple examples in this repository. The basic idea is:

  1. Load the plugin rule: load("@build_stack_rules_proto//:plugin.bzl", "proto_plugin").
  2. Define the rule, giving it a name, options (not mandatory), tool, and outputs.
  3. tool is a label that refers to the binary executable for the plugin itself.
  4. outputs is a list of strings that predicts the pattern of files generated by the plugin. Specifying outputs is the only attribute that requires much mental effort.

Contributing

Contributions welcome; please create Issues or GitHub pull requests.

ProtoInfo provider

A few notes about the 'proto' provider. Bazel initially implemented providers using simple strings. For example, one could write a rule that depends on proto_library rules as follows:

my_custom_rule = rule(
    implementation = my_custom_rule_impl,
    attrs = {
        "deps": attr.label_list(
            doc = "proto_library dependencies",
            mandatory = True,
            providers = ["proto"],
        ),
    }
) 

We can then collect a list of the provided "info" objects as follows:

def my_custom_rule_impl(ctx):
    # list<ProtoInfo> A list of ProtoInfo values
    deps = [ dep.proto for dep in ctx.attr.deps ]

This style of provider usage is now considered "legacy". As of bazel 0.22.0 (Jan 2019) 4817df, the ProtoInfo provider was introduced. Rules using this provider should be changed to:

my_custom_rule = rule(
    implementation = my_custom_rule_impl,
    attrs = {
        "deps": attr.label_list(
            doc = "proto_library dependencies",
            mandatory = True,
            providers = [ProtoInfo],
        ),
 
def my_custom_rule_impl(ctx):
    # <list<ProtoInfo>> A list of ProtoInfo
    deps = [ dep[ProtoInfo] for dep in ctx.attr.deps ]

A flag to toggle availability of the "legacy" provider was introduced in e3f4ce as --incompatible_disable_legacy_proto_provider. Therefore, if you still want the legacy behavior specify --incompatible_disable_legacy_proto_provider=false.

Additional reading:

You can’t perform that action at this time.