RBE / docker sandbox cannot find protobuf dependencies

[lizan]

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

bazel build --config=docker-clang //source/exe:envoy-static from latest master of https://github.com/envoyproxy/envoy.

with bazelisk last_green or last_downstream_green.

This worked well in 0.28.0 / 0.28.1 but not latest master when I was trying out #8934. Without RBE / docker sandbox it works fine too.

What operating system are you running Bazel on?

Ubuntu 16.04

What's the output of bazel info release?

development vresion

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

bazelisk with latest_green

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/lizan/envoy.git
c586af9d3f5940acc73d6a5fd0fdc2c521b6243d
c586af9d3f5940acc73d6a5fd0fdc2c521b6243d

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

SUBCOMMAND: # @envoy_api//envoy/api/v2/core:grpc_service [action 'Generating Descriptor Set proto_library @envoy_api//envoy/api/v2/core:grpc_service']
(cd /home/lizan/.cache/bazel/_bazel_lizan/3a89de24641607fe25c5d5d5ad68d05f/execroot/envoy && \
  exec env - \
    BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 \
    BAZEL_LINKLIBS=-l%:libstdc++.a \
    BAZEL_LINKOPTS=-lm:-static-libgcc \
    CC=clang \
    CXX=clang++ \
    PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/llvm-8/bin \
  bazel-out/host/bin/external/com_google_protobuf/protoc '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/any_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/duration_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/empty_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/struct_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto' '--proto_path=external/com_google_googleapis' '--proto_path=external/com_github_gogo_protobuf' '--proto_path=external/com_envoyproxy_protoc_gen_validate' '--proto_path=external/envoy_api' '--descriptor_set_out=bazel-out/k8-fastbuild/bin/external/envoy_api/envoy/api/v2/core/grpc_service-descriptor-set.proto.bin' '-Ienvoy/api/v2/core/grpc_service.proto=external/envoy_api/envoy/api/v2/core/grpc_service.proto' '-Ienvoy/api/v2/core/base.proto=external/envoy_api/envoy/api/v2/core/base.proto' '-Ienvoy/api/v2/core/http_uri.proto=external/envoy_api/envoy/api/v2/core/http_uri.proto' '-Igoogle/protobuf/any.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/any_proto/google/protobuf/any.proto' '-Igoogle/protobuf/descriptor.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto/google/protobuf/descriptor.proto' '-Igoogle/protobuf/duration.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/duration_proto/google/protobuf/duration.proto' '-Igoogle/protobuf/empty.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/empty_proto/google/protobuf/empty.proto' '-Igoogle/protobuf/struct.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/struct_proto/google/protobuf/struct.proto' '-Igoogle/protobuf/timestamp.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto' '-Igoogle/protobuf/wrappers.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto/google/protobuf/wrappers.proto' '-Igoogle/api/http.proto=external/com_google_googleapis/google/api/http.proto' '-Igoogle/api/annotations.proto=external/com_google_googleapis/google/api/annotations.proto' '-Igoogle/rpc/status.proto=external/com_google_googleapis/google/rpc/status.proto' '-Igogoproto/gogo.proto=external/com_github_gogo_protobuf/gogoproto/gogo.proto' '-Ivalidate/validate.proto=external/com_envoyproxy_protoc_gen_validate/validate/validate.proto' '-Ienvoy/type/percent.proto=external/envoy_api/envoy/type/percent.proto' --direct_dependencies envoy/api/v2/core/base.proto:google/protobuf/any.proto:google/protobuf/descriptor.proto:google/protobuf/duration.proto:google/protobuf/empty.proto:google/protobuf/struct.proto:google/protobuf/timestamp.proto:google/protobuf/wrappers.proto:google/api/http.proto:google/api/annotations.proto:google/rpc/status.proto:gogoproto/gogo.proto:validate/validate.proto:envoy/api/v2/core/grpc_service.proto '--direct_dependencies_violation_msg=%s is imported, but @envoy_api//envoy/api/v2/core:grpc_service doesn'\''t directly depend on a proto_library that '\''srcs'\'' it.' external/envoy_api/envoy/api/v2/core/grpc_service.proto)
ERROR: /home/lizan/.cache/bazel/_bazel_lizan/3a89de24641607fe25c5d5d5ad68d05f/external/com_google_googleapis/google/devtools/cloudtrace/v2/BUILD.bazel:162:1: Action external/com_google_googleapis/google/devtools/cloudtrace/v2/trace.grpc.pb.h failed (Exit 1) protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/grpc_cpp_plugin' ... (remaining 14 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
bazel-out/k8-fastbuild/bin/external/com_google_googleapis/external/com_google_googleapis: warning: directory does not exist.
google/protobuf/descriptor.proto: File not found.
google/api/annotations.proto:20:1: Import "google/protobuf/descriptor.proto" was not found or had errors.
google/api/annotations.proto:28:8: "google.protobuf.MethodOptions" is not defined.
google/protobuf/timestamp.proto: File not found.
google/protobuf/wrappers.proto: File not found.
google/protobuf/any.proto: File not found.
google/rpc/status.proto:19:1: Import "google/protobuf/any.proto" was not found or had errors.
google/rpc/status.proto:93:12: "google.protobuf.Any" is not defined.
google/devtools/cloudtrace/v2/trace.proto:19:1: Import "google/api/annotations.proto" was not found or had errors.
google/devtools/cloudtrace/v2/trace.proto:20:1: Import "google/protobuf/timestamp.proto" was not found or had errors.
google/devtools/cloudtrace/v2/trace.proto:21:1: Import "google/protobuf/wrappers.proto" was not found or had errors.
google/devtools/cloudtrace/v2/trace.proto:22:1: Import "google/rpc/status.proto" was not found or had errors.
google/devtools/cloudtrace/v2/trace.proto:101:5: "google.protobuf.Timestamp" is not defined.
google/devtools/cloudtrace/v2/trace.proto:203:3: "google.protobuf.Timestamp" is not defined.
google/devtools/cloudtrace/v2/trace.proto:208:3: "google.protobuf.Timestamp" is not defined.
google/devtools/cloudtrace/v2/trace.proto:225:3: "google.rpc.Status" is not defined.
google/devtools/cloudtrace/v2/trace.proto:231:3: "google.protobuf.BoolValue" is not defined.
google/devtools/cloudtrace/v2/trace.proto:235:3: "google.protobuf.Int32Value" is not defined.
Target //source/exe:envoy-static failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/lizan/.cache/bazel/_bazel_lizan/3a89de24641607fe25c5d5d5ad68d05f/external/com_google_googleapis/google/devtools/cloudtrace/v2/BUILD.bazel:162:1 Action external/com_google_googleapis/google/devtools/cloudtrace/v2/trace.grpc.pb.h failed (Exit 1) protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/grpc_cpp_plugin' ... (remaining 14 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox

[Qinusty]

Also seeing this issue in https://gitlab.com/remote-apis-testing/remote-apis-testing/ using alternative RE-API server implementations. Similar failures can be seen with buildbarn and buildfarm.

Bazel is installed via apt as follows:

RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt testing jdk1.8" | tee /etc/apt/sources.list.d/bazel.list
RUN curl https://bazel.build/bazel-release.pub.gpg | apt-key add -
RUN apt update && apt -yq install bazel

$ Bazel info release
release 0.29.0rc2

[buchgr]

The culprit is bcc9812. I don't know why (yet). @lberki

[buchgr]

@lberki steps to reproduce:

1. Build a Bazel binary from master
2. git clone https://github.com/envoyproxy/envoy.git
3. Apply this patch

diff --git a/bazel/toolchains/rbe_toolchains_config.bzl b/bazel/toolchains/rbe_toolchains_config.bzl
index fd7210db1..490a536f4 100644
--- a/bazel/toolchains/rbe_toolchains_config.bzl
+++ b/bazel/toolchains/rbe_toolchains_config.bzl
@@ -64,7 +64,7 @@ def _envoy_rbe_toolchain(name, env, toolchain_config_spec_name):
         repository = _ENVOY_BUILD_IMAGE_REPOSITORY,
         toolchain_config_spec_name = toolchain_config_spec_name,
         toolchain_config_suite_spec = _TOOLCHAIN_CONFIG_SUITE_SPEC,
-        use_checked_in_confs = "Force",
+        use_checked_in_confs = "False",
     )
 
 def rbe_toolchains_config():

4. bazel-from-master build --config=docker-clang //source/exe:envoy-static

[lberki]

I don't think I'll be able to do anything about this today, but I at least managed to reproduce it. Kinda weird, because I remember fixing the exact same breakage in Envoy when I submitted the culprit change.

[buchgr]

@lberki do you consider this a release blocker for 0.29.0?

[lberki]

yes (at least until I figure out why this happens)

[lberki]

Faster reproduction:

bazel build --config=docker-clang @com_google_googleapis//google/devtools/cloudtrace/v2:_trace_cc_grpc_grpc_codegen

[lberki]

Welp. It appears that the _generate_cc rule of gRPC is incompatible with Bazel 0.29. In particular, this line:

https://github.com/grpc/grpc/blob/master/bazel/generate_cc.bzl#L114

produces the wrong list of --proto_path flags. Both 0.28 and 0.29 say:

    '--proto_path=external/com_google_googleapis' \
    '--proto_path=external/com_google_googleapis' \
    '--proto_path=external/com_google_googleapis' \
    '--proto_path=external/com_google_googleapis' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=external/com_google_googleapis' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_googleapis/external/com_google_googleapis'

but 0.29 puts e.g. timestamp.proto under bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto, which gRPC is not prepared for.

[lberki]

The good news is that I have a patch:

--- a/bazel/protobuf.bzl	2019-08-06 13:50:24.953923655 +0200
+++ b/bazel/protobuf.bzl	2019-08-06 13:49:49.990103380 +0200
@@ -59,6 +59,13 @@
 def _get_include_directory(include):
     directory = include.path
     prefix_len = 0
+
+    virtual_imports = "/_virtual_imports/"
+    if not include.is_source and virtual_imports in include.path:
+        root, relative = include.path.split(virtual_imports, 2)
+        result = root + virtual_imports + relative.split("/", 1)[0]
+        return result
+
     if not include.is_source and directory.startswith(include.root.path):
         prefix_len = len(include.root.path) + 1

The bad news this is for gRPC, so I'll have to figure out how to up-integrate it into Envoy. Or else leave it to Someone Else™.

[lberki]

Update: patch to gRPC sent out:

grpc/grpc#19860

[lberki]

Filed bug on Envoy:

envoyproxy/envoy#7835

There is also a pull request to gRPC, so I consider my work done here.