Bazel CI: Projects are failing on RBE platform

[meteorcloudy]

https://buildkite.com/bazel/bazel-with-downstream-projects-bazel/builds/607#b722acb8-b6a6-413f-87b6-eed40d652fe1

ERROR: No toolchain found for cpu 'k8'. Valid toolchains are: [
--
  | stub_armeabi-v7a: --cpu='armeabi-v7a' --compiler='compiler',
  | linux_gnu_x86: --cpu='k8' --compiler='clang',
  | msys_x64_mingw: --cpu='x64_windows' --compiler='mingw-gcc',
  | msvc_x64: --cpu='x64_windows' --compiler='msvc-cl',
  | ]

@mhlopko Any idea what caused this error?
FYI @nlopezgi @xingao267

[nlopezgi]

This is likely caused by using an old toolchain config with a newer bazel version which fails due to https://docs.bazel.build/versions/master/skylark/backward-compatibility.html#disallow-using-crosstool-to-select-the-cc_toolchain-label. Please double check the WORKSPACE pin to the bazel-toolchains repo for the project that is failing and make sure its at the latest value published here https://releases.bazel.build/bazel-toolchains.html and that they are using "latest" (https://github.com/bazelbuild/bazel-toolchains/blob/master/configs/ubuntu16_04_clang/latest/BUILD) targets for their toolchain config flags (platforms, crosstool, javabase) (if this is part of a buildkite rbe CI run which should already be using the "latest" toolchain targets, just making sure the failing project is pinned to the latest bazel-toolchains release should fix this)

[nlopezgi]

Note if this is failing with Bazel head and the pin to WORKSPACE is already at the latest bazel-toolchains release, then it means a recent change in toolchain selection is breaking for using toolchain configs for the latest bazel release. If so, this change will be problematic to release w/o proper guarding via --incompatible_xx flags.

[meteorcloudy]

@mhlopko Can you look into this?

[hlopko]

Looking into it.

CC @vladmos FYI

[hlopko]

All PRs uploaded, @vladmos pls close the issue once they are all submitted (currently only rules_nodejs is pending) Thanks all!

[vladmos]

All PRs are submitted.