Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RBE / docker sandbox cannot find protobuf dependencies #9032

Closed
lizan opened this issue Aug 1, 2019 · 11 comments
Closed

RBE / docker sandbox cannot find protobuf dependencies #9032

lizan opened this issue Aug 1, 2019 · 11 comments
Assignees
Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team

Comments

@lizan
Copy link

lizan commented Aug 1, 2019

Description of the problem / feature request:

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

bazel build --config=docker-clang //source/exe:envoy-static from latest master of https://github.com/envoyproxy/envoy.

with bazelisk last_green or last_downstream_green.

This worked well in 0.28.0 / 0.28.1 but not latest master when I was trying out #8934. Without RBE / docker sandbox it works fine too.

What operating system are you running Bazel on?

Ubuntu 16.04

What's the output of bazel info release?

development vresion

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

bazelisk with latest_green

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/lizan/envoy.git
c586af9d3f5940acc73d6a5fd0fdc2c521b6243d
c586af9d3f5940acc73d6a5fd0fdc2c521b6243d

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

SUBCOMMAND: # @envoy_api//envoy/api/v2/core:grpc_service [action 'Generating Descriptor Set proto_library @envoy_api//envoy/api/v2/core:grpc_service']
(cd /home/lizan/.cache/bazel/_bazel_lizan/3a89de24641607fe25c5d5d5ad68d05f/execroot/envoy && \
  exec env - \
    BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1 \
    BAZEL_LINKLIBS=-l%:libstdc++.a \
    BAZEL_LINKOPTS=-lm:-static-libgcc \
    CC=clang \
    CXX=clang++ \
    PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/llvm-8/bin \
  bazel-out/host/bin/external/com_google_protobuf/protoc '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/any_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/duration_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/empty_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/struct_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto' '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto' '--proto_path=external/com_google_googleapis' '--proto_path=external/com_github_gogo_protobuf' '--proto_path=external/com_envoyproxy_protoc_gen_validate' '--proto_path=external/envoy_api' '--descriptor_set_out=bazel-out/k8-fastbuild/bin/external/envoy_api/envoy/api/v2/core/grpc_service-descriptor-set.proto.bin' '-Ienvoy/api/v2/core/grpc_service.proto=external/envoy_api/envoy/api/v2/core/grpc_service.proto' '-Ienvoy/api/v2/core/base.proto=external/envoy_api/envoy/api/v2/core/base.proto' '-Ienvoy/api/v2/core/http_uri.proto=external/envoy_api/envoy/api/v2/core/http_uri.proto' '-Igoogle/protobuf/any.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/any_proto/google/protobuf/any.proto' '-Igoogle/protobuf/descriptor.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto/google/protobuf/descriptor.proto' '-Igoogle/protobuf/duration.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/duration_proto/google/protobuf/duration.proto' '-Igoogle/protobuf/empty.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/empty_proto/google/protobuf/empty.proto' '-Igoogle/protobuf/struct.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/struct_proto/google/protobuf/struct.proto' '-Igoogle/protobuf/timestamp.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto' '-Igoogle/protobuf/wrappers.proto=bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto/google/protobuf/wrappers.proto' '-Igoogle/api/http.proto=external/com_google_googleapis/google/api/http.proto' '-Igoogle/api/annotations.proto=external/com_google_googleapis/google/api/annotations.proto' '-Igoogle/rpc/status.proto=external/com_google_googleapis/google/rpc/status.proto' '-Igogoproto/gogo.proto=external/com_github_gogo_protobuf/gogoproto/gogo.proto' '-Ivalidate/validate.proto=external/com_envoyproxy_protoc_gen_validate/validate/validate.proto' '-Ienvoy/type/percent.proto=external/envoy_api/envoy/type/percent.proto' --direct_dependencies envoy/api/v2/core/base.proto:google/protobuf/any.proto:google/protobuf/descriptor.proto:google/protobuf/duration.proto:google/protobuf/empty.proto:google/protobuf/struct.proto:google/protobuf/timestamp.proto:google/protobuf/wrappers.proto:google/api/http.proto:google/api/annotations.proto:google/rpc/status.proto:gogoproto/gogo.proto:validate/validate.proto:envoy/api/v2/core/grpc_service.proto '--direct_dependencies_violation_msg=%s is imported, but @envoy_api//envoy/api/v2/core:grpc_service doesn'\''t directly depend on a proto_library that '\''srcs'\'' it.' external/envoy_api/envoy/api/v2/core/grpc_service.proto)
ERROR: /home/lizan/.cache/bazel/_bazel_lizan/3a89de24641607fe25c5d5d5ad68d05f/external/com_google_googleapis/google/devtools/cloudtrace/v2/BUILD.bazel:162:1: Action external/com_google_googleapis/google/devtools/cloudtrace/v2/trace.grpc.pb.h failed (Exit 1) protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/grpc_cpp_plugin' ... (remaining 14 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
bazel-out/k8-fastbuild/bin/external/com_google_googleapis/external/com_google_googleapis: warning: directory does not exist.
google/protobuf/descriptor.proto: File not found.
google/api/annotations.proto:20:1: Import "google/protobuf/descriptor.proto" was not found or had errors.
google/api/annotations.proto:28:8: "google.protobuf.MethodOptions" is not defined.
google/protobuf/timestamp.proto: File not found.
google/protobuf/wrappers.proto: File not found.
google/protobuf/any.proto: File not found.
google/rpc/status.proto:19:1: Import "google/protobuf/any.proto" was not found or had errors.
google/rpc/status.proto:93:12: "google.protobuf.Any" is not defined.
google/devtools/cloudtrace/v2/trace.proto:19:1: Import "google/api/annotations.proto" was not found or had errors.
google/devtools/cloudtrace/v2/trace.proto:20:1: Import "google/protobuf/timestamp.proto" was not found or had errors.
google/devtools/cloudtrace/v2/trace.proto:21:1: Import "google/protobuf/wrappers.proto" was not found or had errors.
google/devtools/cloudtrace/v2/trace.proto:22:1: Import "google/rpc/status.proto" was not found or had errors.
google/devtools/cloudtrace/v2/trace.proto:101:5: "google.protobuf.Timestamp" is not defined.
google/devtools/cloudtrace/v2/trace.proto:203:3: "google.protobuf.Timestamp" is not defined.
google/devtools/cloudtrace/v2/trace.proto:208:3: "google.protobuf.Timestamp" is not defined.
google/devtools/cloudtrace/v2/trace.proto:225:3: "google.rpc.Status" is not defined.
google/devtools/cloudtrace/v2/trace.proto:231:3: "google.protobuf.BoolValue" is not defined.
google/devtools/cloudtrace/v2/trace.proto:235:3: "google.protobuf.Int32Value" is not defined.
Target //source/exe:envoy-static failed to build
Use --verbose_failures to see the command lines of failed build steps.
ERROR: /home/lizan/.cache/bazel/_bazel_lizan/3a89de24641607fe25c5d5d5ad68d05f/external/com_google_googleapis/google/devtools/cloudtrace/v2/BUILD.bazel:162:1 Action external/com_google_googleapis/google/devtools/cloudtrace/v2/trace.grpc.pb.h failed (Exit 1) protoc failed: error executing command bazel-out/host/bin/external/com_google_protobuf/protoc '--plugin=protoc-gen-PLUGIN=bazel-out/host/bin/external/com_github_grpc_grpc/grpc_cpp_plugin' ... (remaining 14 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
@irengrig irengrig added P0 This is an emergency and more important than other current work. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team labels Aug 2, 2019
@buchgr buchgr added P1 I'll work on this now. (Assignee required) and removed P0 This is an emergency and more important than other current work. (Assignee required) labels Aug 2, 2019
@Qinusty
Copy link
Contributor

Qinusty commented Aug 5, 2019

Also seeing this issue in https://gitlab.com/remote-apis-testing/remote-apis-testing/ using alternative RE-API server implementations. Similar failures can be seen with buildbarn and buildfarm.

Bazel is installed via apt as follows:

RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt testing jdk1.8" | tee /etc/apt/sources.list.d/bazel.list
RUN curl https://bazel.build/bazel-release.pub.gpg | apt-key add -
RUN apt update && apt -yq install bazel
$ Bazel info release
release 0.29.0rc2

@buchgr
Copy link
Contributor

buchgr commented Aug 5, 2019

The culprit is bcc9812. I don't know why (yet). @lberki

@buchgr
Copy link
Contributor

buchgr commented Aug 6, 2019

@lberki steps to reproduce:

  1. Build a Bazel binary from master
  2. git clone https://github.com/envoyproxy/envoy.git
  3. Apply this patch
diff --git a/bazel/toolchains/rbe_toolchains_config.bzl b/bazel/toolchains/rbe_toolchains_config.bzl
index fd7210db1..490a536f4 100644
--- a/bazel/toolchains/rbe_toolchains_config.bzl
+++ b/bazel/toolchains/rbe_toolchains_config.bzl
@@ -64,7 +64,7 @@ def _envoy_rbe_toolchain(name, env, toolchain_config_spec_name):
         repository = _ENVOY_BUILD_IMAGE_REPOSITORY,
         toolchain_config_spec_name = toolchain_config_spec_name,
         toolchain_config_suite_spec = _TOOLCHAIN_CONFIG_SUITE_SPEC,
-        use_checked_in_confs = "Force",
+        use_checked_in_confs = "False",
     )
 
 def rbe_toolchains_config():
  1. bazel-from-master build --config=docker-clang //source/exe:envoy-static

@buchgr buchgr assigned lberki and unassigned buchgr Aug 6, 2019
@lberki
Copy link
Contributor

lberki commented Aug 6, 2019

I don't think I'll be able to do anything about this today, but I at least managed to reproduce it. Kinda weird, because I remember fixing the exact same breakage in Envoy when I submitted the culprit change.

@buchgr
Copy link
Contributor

buchgr commented Aug 6, 2019

@lberki do you consider this a release blocker for 0.29.0?

@lberki
Copy link
Contributor

lberki commented Aug 6, 2019

yes (at least until I figure out why this happens)

@lberki
Copy link
Contributor

lberki commented Aug 6, 2019

Faster reproduction:

bazel build --config=docker-clang @com_google_googleapis//google/devtools/cloudtrace/v2:_trace_cc_grpc_grpc_codegen

@lberki
Copy link
Contributor

lberki commented Aug 6, 2019

Welp. It appears that the _generate_cc rule of gRPC is incompatible with Bazel 0.29. In particular, this line:

https://github.com/grpc/grpc/blob/master/bazel/generate_cc.bzl#L114

produces the wrong list of --proto_path flags. Both 0.28 and 0.29 say:

    '--proto_path=external/com_google_googleapis' \
    '--proto_path=external/com_google_googleapis' \
    '--proto_path=external/com_google_googleapis' \
    '--proto_path=external/com_google_googleapis' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=external/com_google_googleapis' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_protobuf' \
    '--proto_path=bazel-out/k8-fastbuild/bin/external/com_google_googleapis/external/com_google_googleapis'

but 0.29 puts e.g. timestamp.proto under bazel-out/k8-fastbuild/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto/google/protobuf/timestamp.proto, which gRPC is not prepared for.

@lberki
Copy link
Contributor

lberki commented Aug 6, 2019

The good news is that I have a patch:

--- a/bazel/protobuf.bzl	2019-08-06 13:50:24.953923655 +0200
+++ b/bazel/protobuf.bzl	2019-08-06 13:49:49.990103380 +0200
@@ -59,6 +59,13 @@
 def _get_include_directory(include):
     directory = include.path
     prefix_len = 0
+
+    virtual_imports = "/_virtual_imports/"
+    if not include.is_source and virtual_imports in include.path:
+        root, relative = include.path.split(virtual_imports, 2)
+        result = root + virtual_imports + relative.split("/", 1)[0]
+        return result
+
     if not include.is_source and directory.startswith(include.root.path):
         prefix_len = len(include.root.path) + 1

The bad news this is for gRPC, so I'll have to figure out how to up-integrate it into Envoy. Or else leave it to Someone Else™.

@lberki
Copy link
Contributor

lberki commented Aug 6, 2019

Update: patch to gRPC sent out:

grpc/grpc#19860

@lberki
Copy link
Contributor

lberki commented Aug 6, 2019

Filed bug on Envoy:

envoyproxy/envoy#7835

There is also a pull request to gRPC, so I consider my work done here.

@lberki lberki closed this as completed Aug 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P1 I'll work on this now. (Assignee required) team-Remote-Exec Issues and PRs for the Execution (Remote) team
Projects
None yet
Development

No branches or pull requests

5 participants