incompatible_allow_python_version_transitions: Python version is no longer "sticky"

[brandjon]

Flag: --incompatible_allow_python_version_transitions
Available since: 0.23
Will be flipped in: either 0.24 or 0.25, depending on migration progress
Tracking issue: #6583
Design doc
Corresponding syntactic change: #7308

Motivation

The Python mode behaved in a confusing and error-prone way, and didn't support important use cases like a Python 3 binary depending on a Python 2 one in its data attribute.

See the design doc for more details.

Note: This incompatible change affects the mechanism that Bazel uses to decide whether a target should be built for Python 2 or 3. However, due to #4815, even when a target is built for Python 3 it may still execute under a Python 2 interpreter. See the workaround using py_runtime / --python_top in #4815 (comment).

Change

This makes the Python version have only two possible states: PY2 and PY3. The version can change whenever an executable Python rule (py_binary or py_test) is encountered, even if the version was previously set by another target or by a top-level flag.

In addition, srcs_version validation is done at the level of the executable target, rather than in each py_library. This makes it so you can do bazel build //... without getting errors from PY2-only or PY3-only libraries.

This change does not remove any syntax. See --incompatible_remove_old_python_version_api for that.

Note: This change does not affect targets built in the host configuration, such as targets in genrule's tools attribute, or user-defined rules that set cfg = host on an attribute. This is a consequence of technical limitations on the host configuration to be addressed in future work. In the meantime, you can set a Python version to use globally for all host-configured targets by setting --host_force_python=[PY2 | PY3].

Migration

If you were relying on --force_python to control your Python mode, that won't work anymore since py_binary will set its own mode. Use the python_version attribute (formerly default_python_version) on py_binary instead.

Using multiple Python versions in the same build can lead to action conflicts if you test the python version by select()-ing on "force_python". To avoid this, you should instead select on @bazel_tools//tools/python:python_version. See here. You can ensure that your build doesn't select on "force_python" by enabling --incompatible_remove_old_python_version_api.

Another type of action conflict can arise due to #7655 when a cc_library is data-depended on by a py_binary. This kind of conflict can be avoided by enabling --incompatible_remove_old_python_version_api. If you're not able to migrate your build to enable that flag yet, you can also avoid this particular conflict by explicitly passing a value for the --force_python flag that matches the value passed for --python_version (or the default value if --python_version is not passed).

If there's a srcs_version conflict in your build, you will no longer see an error at the py_library that introduces the conflict. Instead you'll see it in the py_binary or py_test that transitively depends on the library. The error message will not identify which library introduced the version constraint. To find it, you can run the aspect @bazel_tools//tools/python:srcs_version.bzl%find_requirements as follows:

bazel build <your target> \
    --aspects=@bazel_tools//tools/python:srcs_version.bzl%find_requirements \
    --output_groups=pyversioninfo

This will build a [...]-pyversioninfo.txt file explaining what dependency requires what Python version. You can run it directly on the py_binary or py_test that failed due to a version conflict.

[katre]

This issue was tagged as "breaking-change-0.24" but does not appear ready to be flipped in the 0.24.0 release. If this is incorrect please comment on that issue and discuss with me.

[brandjon]

Now targeting 0.25.

[brandjon]

Note the updated paragraph in Migration instructions above to account for #7655.

[brandjon]

According to a recent run of the bazelisk CI, the only downstream failures due to --incompatible_allow_python_version_transitions are due to action conflicts in C++ rules, which should go away when --incompatible_remove_old_python_version_api is enabled.

[brandjon]

I'm submitting the flag flip to remove the old version api now. That should give us a clean run of the bazelisk CI for this flag tomorrow.

[brandjon]

The bazelisk run isn't clean because closing the migration bug for --incompatible_remove_old_python_version_api doesn't cause bazelisk to run its targets with that flag enabled -- only a new Bazel release containing the flag flip can do that. Since both of these flag flips are happening in 0.25, we can't expect a clean Bazelisk CI.

At this point I think it is in practice safe to flip the flag, but in principle we're still blocked by #7808. I'd like to fix that bug first. Shouldn't be a problem for making the 0.25 cut.

[brandjon]

Submitting the flag flip because #7808 is fixed.