[Bazel] Build error from source

[acxz]

What is the problem?

Have the following build error when building from source using bazel 4.0.

DEBUG: /home/acxz/vcs/git/github/acxz/pkgbuilds/python-ray/src/ray-ray-1.2.0/bazel/ray_deps_setup.bzl:63:14: No implicit mirrors used because urls were explicitly provided
ERROR: /home/acxz/.cache/bazel/_bazel_acxz/efe9f06255fafeaf8887ca9ca2d64c2e/external/com_google_protobuf/BUILD:873:21: in proto_lang_toolchain rule @com_google_protobuf//:cc_toolchain: '@com_google_protobuf//:cc_toolchain' does not have mandatory provider 'ProtoInfo'.
INFO: Repository boost instantiated at:
  /home/acxz/vcs/git/github/acxz/pkgbuilds/python-ray/src/ray-ray-1.2.0/WORKSPACE:5:15: in <toplevel>
  /home/acxz/vcs/git/github/acxz/pkgbuilds/python-ray/src/ray-ray-1.2.0/bazel/ray_deps_setup.bzl:134:22: in ray_deps_setup
  /home/acxz/vcs/git/github/acxz/pkgbuilds/python-ray/src/ray-ray-1.2.0/bazel/ray_deps_setup.bzl:74:24: in auto_http_archive
Repository rule http_archive defined at:
  /home/acxz/.cache/bazel/_bazel_acxz/efe9f06255fafeaf8887ca9ca2d64c2e/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: Analysis of target '//:ray_pkg' failed; build aborted: Analysis of target '@com_google_protobuf//:cc_toolchain' failed
INFO: Elapsed time: 1.303s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (7 packages loaded, 247 targets configured)
    Fetching @cython; Restarting.

Ray version and other system information (Python version, TensorFlow version, OS):
Bazel: 4.0

Reproduction (REQUIRED)

N/A
Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

If the code snippet cannot be run by itself, the issue will be closed with "needs-repro-script".

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[acxz]

Looks like the problem stems from bazelbuild/bazel#12887

[rkooo567]

The fix sounds quite simple. Can you try that out and see if the issue is resolved in your local branch?

[acxz]

I gave it a shot, but I think it is a bit more complicated than at first glance (or maybe I'm just not able to read the error messages properly). I think I made some progress by upgrading

ray/bazel/ray_deps_setup.bzl

Line 248 in 019d84a

url = "https://github.com/rules-proto-grpc/rules_proto_grpc/archive/a74fef39c5fe636580083545f76d1eab74f6450d.tar.gz",

but am still having the issue, most likely do to another dependency.

[paulfryzel]

From bazelbuild/bazel#12887 (comment):

diff --git a/bazel/ray_deps_build_all.bzl b/bazel/ray_deps_build_all.bzl
index a81480664..5cc79aec7 100644
--- a/bazel/ray_deps_build_all.bzl
+++ b/bazel/ray_deps_build_all.bzl
@@ -6,6 +6,7 @@ load("@com_github_jupp0r_prometheus_cpp//bazel:repositories.bzl", "prometheus_cp
 load("@com_github_grpc_grpc//third_party/py:python_configure.bzl", "python_configure")
 load("@com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
 load("@rules_proto_grpc//:repositories.bzl", "rules_proto_grpc_toolchains")
+load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
 
 
 def ray_deps_build_all():
@@ -17,3 +18,5 @@ def ray_deps_build_all():
   python_configure(name = "local_config_python")
   grpc_deps()
   rules_proto_grpc_toolchains()
+  rules_proto_dependencies()
+  rules_proto_toolchains()
\ No newline at end of file
diff --git a/bazel/ray_deps_setup.bzl b/bazel/ray_deps_setup.bzl
index c398dd15d..20b6bbda5 100644
--- a/bazel/ray_deps_setup.bzl
+++ b/bazel/ray_deps_setup.bzl
@@ -265,3 +265,10 @@ def ray_deps_setup():
         urls = ["https://github.com/census-instrumentation/opencensus-proto/archive/v0.3.0.tar.gz"],
         sha256 = "b7e13f0b4259e80c3070b583c2f39e53153085a6918718b1c710caf7037572b0",
     )
+
+    auto_http_archive(
+        name = "rules_proto",
+        sha256 = "d8992e6eeec276d49f1d4e63cfa05bbed6d4a26cfe6ca63c972827a0d141ea3b",
+        strip_prefix = "rules_proto-cfdc2fa31879c0aebe31ce7702b1a9c8a4be02d2",
+        url = "https://github.com/bazelbuild/rules_proto/archive/cfdc2fa31879c0aebe31ce7702b1a9c8a4be02d2.tar.gz",
+    )
\ No newline at end of file
diff --git a/python/setup.py b/python/setup.py
index 62c516e9c..b46d9749c 100644
--- a/python/setup.py
+++ b/python/setup.py
@@ -28,7 +28,7 @@ logger = logging.getLogger(__name__)
 # manually.
 
 SUPPORTED_PYTHONS = [(3, 6), (3, 7), (3, 8)]
-SUPPORTED_BAZEL = (3, 2, 0)
+SUPPORTED_BAZEL = (4, 0, 0)
 
 ROOT_DIR = os.path.dirname(__file__)
 BUILD_JAVA = os.getenv("RAY_INSTALL_JAVA") == "1"

... should do it. Verified working build+tests w/ bazel 4.0.0 on macOS 11.2.2, ray at ea7d4c6.

@rkooo567 Is this something you'd be open to a PR on?

[Bam4d]

So did the PR for the above fix, but the CI build fails for other protobuf-version related reasons. I dont know enough about ray build chain to make the right changes in the build to make it work.

It build completely fine locally for me with pip install -e . which is the same thing that happens on the server, So guessing it might be a dependency in a docker image?

EDIT: RAY_INSTALL_JAVA=1 will build the java dependencies also, can reproduce the error using this (and fix it by bumping protobuf to 3.10.0 from 3.8.0 https://stackoverflow.com/questions/58537310/grpc-maven-unable-to-use-oneof-symbol-internaloneofenum-missing)

[Vigilans]

From bazelbuild/bazel#12887 (comment) and bazelbuild/bazel#12887 (comment), update protobuf to latest version also works (built with @acxz 's py39 branch merged):

diff --git a/bazel/ray_deps_setup.bzl b/bazel/ray_deps_setup.bzl
index fcef5edee..df099a67a 100644
--- a/bazel/ray_deps_setup.bzl
+++ b/bazel/ray_deps_setup.bzl
@@ -243,6 +243,12 @@ def ray_deps_setup():
         sha256 = "781fa39693ec2984c71213cd633e9f6589eaaed75e3a9ac413237edec96fd3b9",
     )
 
+    http_archive(
+        name = "com_google_protobuf",
+        strip_prefix = "protobuf-3.15.6",
+        urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.15.6.tar.gz"],
+    )
+
     auto_http_archive(
         name = "rules_proto_grpc",
         url = "https://github.com/rules-proto-grpc/rules_proto_grpc/archive/a74fef39c5fe636580083545f76d1eab74f6450d.tar.gz",