Migrate bazel proto building rules to stackb/rules_proto

[jdwestbro]

This patch migrates the Bazel build to use https://github.com/stackb/rules_proto

This fixes #18256

[thelinuxfoundation]

Thank you for your pull request. Before we can look at your contribution, we need to ensure all contributors are covered by a Contributor License Agreement.

After the following items are addressed, please respond with a new comment here, and the automated system will re-verify.

• User @jdwestbro isn't covered by a CLA. They will need to complete the form at https://identity.linuxfoundation.org/projects/cncf

Regards,
CLA GitHub bot

[jdwestbro]

Associated my github account with @thelinuxfoundation

[nicolasnoble]

I think these are new test failures from this PR: https://source.cloud.google.com/results/invocations/023cddaf-ff1f-4fbe-ace3-720645c3dacb/targets/grpc%2Fcore%2Fpull_request%2Flinux%2Fgrpc_python_bazel_test/log

FAIL: //src/python/grpcio_tests/tests/health_check:health_servicer_test (see /root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/testlogs/src/python/grpcio_tests/tests/health_check/health_servicer_test/test.log)
INFO: From Testing //src/python/grpcio_tests/tests/health_check:health_servicer_test:
==================== Test output for //src/python/grpcio_tests/tests/health_check:health_servicer_test:
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/bin/src/python/grpcio_tests/tests/health_check/health_servicer_test.runfiles/com_github_grpc_grpc/src/python/grpcio_tests/tests/health_check/_health_servicer_test.py", line 21, in <module>
    from grpc_health.v1 import health
ImportError: No module named grpc_health.v1
================================================================================
FAIL: //src/python/grpcio_tests/tests/reflection:_reflection_servicer_test (see /root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/testlogs/src/python/grpcio_tests/tests/reflection/_reflection_servicer_test/test.log)
INFO: From Testing //src/python/grpcio_tests/tests/reflection:_reflection_servicer_test:
==================== Test output for //src/python/grpcio_tests/tests/reflection:_reflection_servicer_test:
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/bin/src/python/grpcio_tests/tests/reflection/_reflection_servicer_test.runfiles/com_github_grpc_grpc/src/python/grpcio_tests/tests/reflection/_reflection_servicer_test.py", line 19, in <module>
    from grpc_reflection.v1alpha import reflection
ImportError: No module named grpc_reflection.v1alpha
================================================================================
FAIL: //src/python/grpcio_tests/tests/interop:_insecure_intraop_test (see /root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/testlogs/src/python/grpcio_tests/tests/interop/_insecure_intraop_test/test.log)
INFO: From Testing //src/python/grpcio_tests/tests/interop:_insecure_intraop_test:
==================== Test output for //src/python/grpcio_tests/tests/interop:_insecure_intraop_test:
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/bin/src/python/grpcio_tests/tests/interop/_insecure_intraop_test.runfiles/com_github_grpc_grpc/src/python/grpcio_tests/tests/interop/_insecure_intraop_test.py", line 19, in <module>
    from src.proto.grpc.testing import test_pb2_grpc
ImportError: cannot import name test_pb2_grpc
================================================================================
FAIL: //src/python/grpcio_tests/tests/interop:_secure_intraop_test (see /root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/testlogs/src/python/grpcio_tests/tests/interop/_secure_intraop_test/test.log)
INFO: From Testing //src/python/grpcio_tests/tests/interop:_secure_intraop_test:
==================== Test output for //src/python/grpcio_tests/tests/interop:_secure_intraop_test:
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/bin/src/python/grpcio_tests/tests/interop/_secure_intraop_test.runfiles/com_github_grpc_grpc/src/python/grpcio_tests/tests/interop/_secure_intraop_test.py", line 19, in <module>
    from src.proto.grpc.testing import test_pb2_grpc
ImportError: cannot import name test_pb2_grpc
================================================================================
FAIL: //src/python/grpcio_tests/tests/channelz:channelz_servicer_test (see /root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/testlogs/src/python/grpcio_tests/tests/channelz/channelz_servicer_test/test.log)
INFO: From Testing //src/python/grpcio_tests/tests/channelz:channelz_servicer_test:
==================== Test output for //src/python/grpcio_tests/tests/channelz:channelz_servicer_test:
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/execroot/com_github_grpc_grpc/bazel-out/k8-fastbuild/bin/src/python/grpcio_tests/tests/channelz/channelz_servicer_test.runfiles/com_github_grpc_grpc/src/python/grpcio_tests/tests/channelz/_channelz_servicer_test.py", line 21, in <module>
    from grpc_channelz.v1 import channelz
ImportError: No module named grpc_channelz.v1

[jdwestbro]

Working on this. Having trouble wrangling the Python tests.

[jdwestbro]

The remaining problem seems to be associated with namespace shadowing related to the code generated using the python_proto_library/python_grpc_library rules:

$ bazel build src/proto/grpc/testing:py_empty_proto
Target //src/proto/grpc/testing:py_empty_proto up-to-date:
bazel-genfiles/src/proto/grpc/testing/py_empty_proto_pb/src/proto/grpc/testing/empty_pb2.py

$ bazel build src/proto/grpc/reflection/v1alpha:py_reflection_proto
Target //src/proto/grpc/reflection/v1alpha:py_reflection_proto up-to-date:
bazel-genfiles/.../py_reflection_proto_pb/src/proto/grpc/reflection/v1alpha/reflection_pb2.py
bazel-genfiles/.../py_reflection_proto_pb/src/proto/grpc/reflection/v1alpha/reflection_pb2_grpc.py

$ bazel test src/python/grpcio_tests/tests/reflection:all
/src/python/grpcio_tests/tests/reflection/_reflection_servicer_test.py", line 26, in
from src.proto.grpc.testing import empty_pb2
ImportError: No module named testing

Looking at the python_grpc_library rule:
https://github.com/stackb/rules_proto/blob/master/python/python_grpc_library.bzl

native.py_library(
    name = name,
    srcs = [name_pb],
    deps = depset(protobuf_requirements + grpc_requirements).to_list(),
    # This magically adds REPOSITORY_NAME/PACKAGE_NAME/{name_pb} to PYTHONPATH
    imports = [name_pb],
    visibility = visibility,
)

So PYTHONPATH is being augmented and one shadows the other. I believe this is a known python2 issue related to the init.py files (which bazel generates automatically).

[jtattermusch]

@jdwestbro I'm not convinced that migrating over to https://github.com/stackb/rules_proto is what we want - what are the benefits? Also this PR is somewhat misleadingly called "fixed bazel build", but it seems to do a lot more that that.

[stale]

This issue/PR has been automatically marked as stale because it has not had any update (including commits, comments, labels, milestones, etc) for 180 days. It will be closed automatically if no further update occurs in 1 day. Thank you for your contributions!

[jtattermusch]

CC @gnossen in case he wants to reopen.

[gnossen]

We ended up going in a different direction. I think this issue is rightly closed.