Problematic package path for generated python files

[ironstein0]

I am in the process of transitioning a project from rules_protobuf to this new library (as suggested) and I seem to have hit a roadblock along the way. I think there is a problem in the way the python packages are consumed, and more importantly, generated. Looking at the Python example, this is how the generated protobuf & grpc libraries are consumed

from routeguide_pb.example.proto import routeguide_pb2
from routeguide_pb.example.proto import routeguide_pb2_grpc

Which seems a little weird to do. Looking at the rules_protobuf way of doing it, it should have been something that looks like

from example.proto import routeguide_pb2
from example.proto import routeguide_pb2_grpc

which seems more natural to me (personally) because you can import the python library as if it was written inside of example/proto without being aware of how/whether it was generated.

To make things worse, I think this leads to a problem with import other python packages/modules. Llisting everything in the sys.path, this is how it looks

.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/pypi__setuptools_40_8_0
.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/pypi__six_1_12_0
.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/pypi__protobuf_3_6_1
.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/pypi__enum34_1_1_6
.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/pypi__grpcio_1_15_0
.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/pypi__futures_3_2_0
.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto/python/example/routeguide/routeguide_pb
.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto/python/example/routeguide
.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto

Looking inside the path in bold above, that is where the generated modules are. The path for these generated modules looks like

.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto/python/example/routeguide/routeguide_pb/example/python/routeguide/routeguide_pb2.py

I do not understand why the files are within example/python/routeguide within this path. When I looked at how rules_proto did this, all the pb2.py and pb2_grpc.py files would be directly within .../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto/python/example/routeguide/routeguide_pb2.py

This also leads to a problem that is not highlighted by this simple python example. I have a this basic example that recreates the problem.

I simply have a proto library under python/example/routeguide/bar/ and I add it as a dependency to python/example/routeguide:server. As per the observation above, this leads to a module generated at the following path

.../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto/python/example/routeguide/routeguide_pb2/python/example/routeguide/bar/

Along with .../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto/python/example/routeguide/routeguide_pb2 in sys.path.

Now, whenever you would like to import some other module from say for example the python/example package, it would collide with the package with the same name in the path mentioned above. In my example, adding another python library python/example/routeguide/foo highlights the problem.

When you run bazel run python/example/routeguide:server, everything runs okay. But, when you run bazel run python/example/routeguide:server_with_bar, it throws an error saying

Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/7fb78493d4016bfee9e4b3025cb7c4f3/execroot/build_stack_rules_proto/bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto/python/example/routeguide/server.py", line 22, in <module> from python.example.routeguide.foo import foo ImportError: No module named foo

This is because it cannot find foo on the path ../bazel-out/k8-fastbuild/bin/python/example/routeguide/server.runfiles/build_stack_rules_proto/python/example/routeguide/routeguide_pb2/python/example/routeguide

[jdwestbro]

I'm having the same/related problem attempting to use these new rules in the gRPC project: grpc/grpc#18317

Specifically the problem manifests when depending on two py_library rules (created via python_grpc_library) that share a common package prefix. In my case:

bazel-genfiles/.../py_empty_proto_pb/src/proto/grpc/testing/empty_pb2.py
bazel-genfiles/.../py_reflection_proto_pb/src/proto/grpc/reflection/v1alpha/reflection_pb2.py

are generated from different python_grpc_library rules. When attempting to import, reflection_pb2 succeeds but empty_pb2 doesn't, with an error message that the testing module doesn't exist.

[pcj]

I'm closing this as the python implementation now uses an aspect which simplifies the imports considerably. Please open new issues as it will involve a different fix.