Cannot import a module scattered in multiple directories

[plule-ansys]

Hello,

When the files of a module are not all in the same directory, then one seem to shadow the others even with correct "imports" arguments.

For example, in this structure, sub1 and sub2 are meant to be abstracted by the import, and the full my_module module is expected to have both a.py and b.py. However, only one of those can be used at the same time.

• sub1: BUILD.bazel (py_library with "imports = ["sub1"])
• sub1/my_module: a.py
• sub2: BUILD.bazel (py_library with "imports = ["sub2"])
• sub2/my_module: b.py

Basic reproduction here.

• bazel run //:main fails with missing module
• However, if I set the PYTHONPATH to "sub1;sub2" (or "sub1:sub2" on Linux), python main.py succeeds.
• If I remove the dependency and import to //sub1:a then it works
• If I remove the dependency and import to //sub2:b then it works

Bazel version: 3.0.0 and 3.3.0
OS: Seen on Windows 10 and Linux
Rules version: 0.0.2
Python version:  Python 3.7.6 and 3.6.3
Actual use case: Mixing generated and non generated files, or generated rules putting the files in a subfolder like with this code generation: rules_proto_grpc

[thundergolfer]

Hello,

Firstly, thanks for the detailed issue and the reproduction.

I'm pretty sure what you're falling prey to here is 'The __init__.py Trap', because the implementation of rules_python adds __init__.py files into modules.

I've cloned your reproduction, and can see that happening here:

ls /private/var/tmp/_bazel_jbelotti/5a85cfef1ef0000000000000000b117361/execroot/__main__/bazel-out/darwin-fastbuild/bin/main.runfiles/__main__/sub1
__init__.py my_module

ls /private/var/tmp/_bazel_jbelotti/5a85cfef1ef0000000000000000b117361/execroot/__main__/bazel-out/darwin-fastbuild/bin/main.runfiles/__main__/sub2/my_module
__init__.py b.py

Your target source inputs don't include those __init__.py files, but they're there.

If I do something 'dirty', and delete those __init__.py files:

rm -f /private/var/tmp/_bazel_jbelotti/5a85cfef1ef0000000000000000b117361/execroot/__main__/bazel-out/darwin-fastbuild/bin/main.runfiles/__main__/sub2/__init__.py
rm -f /private/var/tmp/_bazel_jbelotti/5a85cfef1ef0000000000000000b117361/execroot/__main__/bazel-out/darwin-fastbuild/bin/main.runfiles/__main__/sub1/__init__.py
rm -f /private/var/tmp/_bazel_jbelotti/5a85cfef1ef0000000000000000b117361/execroot/__main__/bazel-out/darwin-fastbuild/bin/main.runfiles/__main__/sub2/my_module/__init__.py
rm -f /private/var/tmp/_bazel_jbelotti/5a85cfef1ef0000000000000000b117361/execroot/__main__/bazel-out/darwin-fastbuild/bin/main.runfiles/__main__/sub1/my_module/__init__.py

I can get the py_binary to execute:

INFO: Analyzed target //:main (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:main up-to-date:
  bazel-bin/main
INFO: Elapsed time: 0.142s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
am a.py
am b.py

I'm aware of this 'added __init__.py' files thing being a headache somewhere else in these Github issues, but I don't have ready-at-hand knowledge of why they're necessary.

[plule-ansys]

Oh thank you for the explanation. I'm discovering the python ecosystem at the same time and would not have guessed.

Based on this, it's very easy to workaround with the options to control the init files creation:

• Either passing "legacy_create_init = False" per py_test or py_binary
• Or passing --incompatible_default_to_explicit_init_py to making it default to False globally.

I'm aware of this 'added init.py' files thing being a headache somewhere else in these Github issues, but I don't have ready-at-hand knowledge of why they're necessary.

I can't tell for sure, but it looks like it's for compatibility with legacy python version.

[groodt]

@plule-ansys

Yes, it is a legacy foot-cannon in Python. If you're able, you probably should set the following in your .bazelrc, which is what we do at Canva.

# Prevent creation of empty __init__.py
# See: https://github.com/bazelbuild/bazel/issues/10076, https://github.com/bazelbuild/bazel/issues/7386
build --incompatible_default_to_explicit_init_py
test --incompatible_default_to_explicit_init_py

[thundergolfer]

@plule-ansys Try the above ☝️ and let me know if it works and if you're happy to close the issue.

[plule-ansys]

I confirm that it's a fix even outside of the toy reproduction. Thanks again, closing.