Add compatible_with and restricted_to parameters to pip_install generated py_library #565
Description
🚀 feature request
Description
The current py_binary and py_library rules are supporting the following attributes: compatible_with and restricted_to. However, the pip_install rule does not support those, which makes it very hard to use as intended.
Note that pip_install already supports the option as a common attribute, but does nothing with it.
Context:
We use a monorepo with many python projects (including many requirements, py_library, py_binary, py_test). We currently run everything with the same python version (3.6), however moving forward we would need a way to handle multiple python versions in order to allow us to upgrade projects one at a time. We would have py_libraries that support multiple python versions and py_binaries that support a single python versions.
Then the goal would be to test everything under all the versions they are compatible, and I can do that with bazel query and running bazel within different virtual envs.
Here is an example:
environment(
name = "python36",
)
environment(
name = "python37",
)
environment_group(
name = "python_versions",
defaults = [],
environments = [
":python36",
":python37",
])
py_library(
name = "my_lib_1",
srcs = ["my_lib_1.py"],
compatible_with = [":python36"],
)
py_library(
name = "my_lib_2",
srcs = ["my_lib_2.py"],
compatible_with = [":python36", ":python37"],
)
py_library(
name = "my_lib_3",
srcs = ["my_lib_3.py"],
compatible_with = [":python37"],
)
py_binary(
name = "my_app_1",
main = "my_app_1.py",
srcs = ["my_app_1.py"],
deps = [":my_lib_1", ":my_lib_2"],
compatible_with = [":python36"],
)
py_binary(
name = "my_app_2",
main = "my_app_2.py",
srcs = ["my_app_2.py"],
deps = [":my_lib_2", ":my_lib_3"],
compatible_with = [":python37"],
)
Then I can build things using the --target_environment parameter:
bazel build //:my_app_1 --target_environment=//:python36
This will fail as expected:
bazel build //:my_app_1 --target_environment=//:python37
with message:
ERROR: This is a restricted-environment build.
//:my_app_1 does not support:
//:python37
Finally I can query which versions are supported:
> bazel query 'attr("compatible_with", //:python36, //...:all)'
//:my_app_1
//:my_lib_2
//:my_lib_1
> bazel query 'attr("compatible_with", //:python37, //...:all)'
//:my_app_2
//:my_lib_3
//:my_lib_2
However I cannot use this with requirements and the pip_install rule. Because all the requirements added will not support any environment and therefore I am unable to build anything that has the compatible_with attribute.
Describe the solution you'd like
I would like the pip_install rule to support the attributes compatible_with and restricted_to. In order to do something like this:
pip_install(
name = "requirements_py36",
requirements = "//:requirements_py36.txt",
compatible_with = [":python36"],
)
pip_install(
name = "requirements_py37",
requirements = "//:requirements_py37.txt",
compatible_with = [":python37"],
)
However this solution in itself isn't enough. If the :python36 value was added to the py_library generated by pip_install, the :python36 target needs to exist within the @requirements_py36 target, I do not know if there is any way of referencing the local target.
Describe alternatives you've considered
It would seem that the latest approach to do that is to use platforms with bazel, however the current python rules don't seem to support that as the compatible_with and restricted_to parameters accept only an environment rule based on the error messages. I wasn't able to get a solution working with Platforms. Additionally environments are not documented anywhere and it was very difficult to understand how to use these in the first place.
I have tried many different way, one possibility way is to set the environment group with default support for all environments. In which case the requirements with pip_install will support python36 and python37, but If I do this I am unable to use the "compatible_with" flag as bazel will think everything is compatible with everything, it will have no effects.
I am able to use a combination of default to all envs + using restricted_to parameter, however it is impractical and I haven't found any good way to run bazel queries to filter out tests compatible with a specific python version.
In general I would also be very interested in hearing about how other people manage to handle multiple python versions? I know a lot of work has been done for python 2 and 3 support in the past years but I haven't found anything about running multiple python 3 versions in the same repo.