RBE worker pool migration

[coeuvre]

Bazel is migrating its RBE worker pool from n1 machines to e2. We created a new dedicated worker pool with n1 machines to support nested virtualization. This PR updates BRE config to target that dedicated worker pool.

[coeuvre]

Test failed on this line: OSError: [Errno 2] No such file or directory.

Would you please help confirm whether this error is related with this PR?

[philwo]

My guess is that the error happens, because the command no longer runs inside the correct container, thus the tool it is trying to run is missing. But I'm not sure yet, why this is the case... looking into it.

[philwo]

I tried in a separate PR to just bump the bazel-toolchains version to 3.4.0 or 3.3.2 and not even that worked (although with a very different error):

• Bazel 3.4.1 with bazel-toolchains 3.4.0: https://buildkite.com/bazel/android-testing/builds/869
• Bazel 3.4.1 with bazel-toolchains 3.3.2: https://buildkite.com/bazel/android-testing/builds/868
• Bazel 3.3.1 with bazel-toolchains 3.3.2: https://buildkite.com/bazel/android-testing/builds/870

They all fail with this error:

undeclared inclusion(s) in rule '@com_google_protobuf//:protobuf_lite':
--
  | this rule is missing dependency declarations for the following files included by 'external/com_google_protobuf/src/google/protobuf/extension_set.cc':
  | '/usr/local/lib/clang/10.0.0/include/stddef.h'
  | '/usr/local/lib/clang/10.0.0/include/stdarg.h'
    ...

This is super weird. :/

[coeuvre]

Anything I can help to move this PR forward?

[philwo]

@jin Could you help us here?

[nick51913]

ในวันที่ พฤ. 6 ส.ค. 2020 23:52 Philipp Wollermann <notifications@github.com> เขียนว่า:

…

@jin <https://github.com/jin> Could you help us here? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#327 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AP7VXD4OOLYHJ362G67N6KLR7LNTTANCNFSM4OZCAF2A> .

[coeuvre]

@jin Friendly ping :)

[philwo]

We’re going ahead with the infrastructure change next week. If something breaks for this project - this PR contains the fix and we’re also happy to help.

[jin]

Just reran the CI pipeline and it looks like all of the tests are failing:

Ubuntu, failing in test step of RBE:

(07:27:09) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/external/remote_java_tools_linux/java_tools/zlib/BUILD.bazel:3:11: Compiling java_tools/zlib/inffast.c [for host] failed: undeclared inclusion(s) in rule '@remote_java_tools_linux//java_tools/zlib:zlib':
--
  | this rule is missing dependency declarations for the following files included by 'java_tools/zlib/inffast.c':
  | '/usr/local/lib/clang/10.0.0/include/stddef.h'
  | '/usr/local/lib/clang/10.0.0/include/__stddef_max_align_t.h'
  | '/usr/local/lib/clang/10.0.0/include/limits.h'
  | '/usr/local/lib/clang/10.0.0/include/stdarg.h'

I remember seeing this problem when the local and remote clang versions don't match, which results in this problem. (Host uses clang 10.0.0, but not the docker image)

macOS and Windows:

(07:23:09) ERROR: While resolving toolchains for target @androidsdk//:apksigner: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
--
  | (07:23:09) ERROR: Analysis of target '//ui/espresso/CustomMatcherSample:CustomMatcherSample' failed; build aborted: No matching toolchains found for types @bazel_tools//tools/cpp:toolchain_type. Maybe --incompatible_use_cc_configure_from_rules_cc has been flipped and there is no default C++ toolchain added in the WORKSPACE file? See https://github.com/bazelbuild/bazel/issues/10134 for details and migration instructions.
  | (07:23:09) INFO: Elapsed time: 5.530s

[philwo]

Possibly due to the outdated baseline of this PR and Bazel on the CI having been upgraded to an incompatible newer version in the meantime? 🤔

@coeuvre Could you rebase this so that we can see the current behavior?

[meteorcloudy]

Ping, #367 is still waiting for this fix ;)

[coeuvre]

Rebased and upgraded bazel toolchain to latest version but still see the same error.

Keep the toolchain untouched works. https://buildkite.com/bazel/android-testing/builds/1183

[jin]

Looks like the change is minimal, just requires adding the machine type?

[coeuvre]

Yes.

[jin]

Thank you for looking into this!