py_library enforces Python 2 for some reason

[abergmeier]

We are not interested in anything lower than Python 3 so we set all our srcs_version to PY3 fpr py_* rules. Still when Analyzing, Bazel complains:

Rule '@foo//:python_foo' need to be converted to Python 2 (not yet implemented).

The error message is quite unhelpful because it does not explain why it "needs" to be converted nor which thing is not yet implemented.

[abergmeier]

Workaround seems to be --force_python=PY3 and --host_force_python=PY3 in command line.

[dslomov]

Need more helpful error message here

[ulfjack]

I don't think py_library enforces anything. :-)

The srcs_version attribute is mainly intended to support migration from python 2 to python 3. The error message is confusing because Bazel thinks you want it to auto-convert the code from python 3 down to python 2, which is the default output.

I think you could just remove all the srcs_version, and it should work just fine.

[abergmeier-dsfishlabs]

@ulfjack Removing the srcs_version I get:

Converting to Python 3: foo.py failed

with it calling bazel-out/host/bin/external/bazel_tools/tools/python/2to3, which is pretty useless since foo.py is a python3 script already.

[ulfjack]

I'm not sure I follow - can you give me more details on how to reproduce the issue?

[abergmeier-dsfishlabs]

@ulfjack I would really love to have a complete description how the different python conversion/flags/attributes are used and in which combination they should produce which output.
Is there anyone at Google who could describe what the logic behind all that is?

[ulfjack]

We've done some work to support a migration from Python 2 to Python 3, and as part of that, the rules can convert both 2to3 and 3to2 (IIRC) if you have appropriate tools. However, I'm not sure why Blaze thinks that those conversions should be triggered. Unfortunately, that code is ~3-4 years old, and nobody working on Bazel right now has the necessary context.

[abergmeier-dsfishlabs]

@ulfjack I think the implementation is flawed. See 2. in #1580. AFAIU it cannot work correctly this way.
The rest of this "weirdness" then stems from doucmentation issue (1.).

[ulfjack]

If by flawed you mean "it doesn't work for all use cases", then I agree.

Where should we start fixing the python rules? What's the biggest pain point?

[abergmeier-dsfishlabs]

• Before anything else I think there should be unit tests, which test all the different python combinations
• I think the workaround currently could be to introduce a flag for disabling the up/downgrading of python files completely (given that nobody seems to be sure how and when it works) at all or alternatively fix the code that it handles correctly.
• Then of course fix documentation.
• Fix referencing python2 at all. Might be safe for macOS but not for Ubuntu 16.04+ or Windows.

[BrendanDrewZippy]

I have also run into this problem. The documentation gives the expectation that if you write python 3 code and set srcs_version='PY3' in your BUILD, everything should "just work". This is not the case.

I think I'm up-to-date. bazel info gives:

INFO: Reading 'startup' options from /Users/bdrew/devel/zippy/.bazelrc: --host_jvm_args=-Xmx2500m --host_jvm_args=-Xms2500m --batch
bazel-bin: /private/var/tmp/_bazel_bdrew/36485b236a991e0e17b0caecf9bb47ca/execroot/zippy/bazel-out/darwin_x86_64-fastbuild/bin
bazel-genfiles: /private/var/tmp/_bazel_bdrew/36485b236a991e0e17b0caecf9bb47ca/execroot/zippy/bazel-out/darwin_x86_64-fastbuild/genfiles
bazel-testlogs: /private/var/tmp/_bazel_bdrew/36485b236a991e0e17b0caecf9bb47ca/execroot/zippy/bazel-out/darwin_x86_64-fastbuild/testlogs
character-encoding: file.encoding = ISO-8859-1, defaultCharset = ISO-8859-1
command_log: /private/var/tmp/_bazel_bdrew/36485b236a991e0e17b0caecf9bb47ca/command.log
committed-heap-size: 2512MB
execution_root: /private/var/tmp/_bazel_bdrew/36485b236a991e0e17b0caecf9bb47ca/execroot/zippy
gc-count: 3
gc-time: 93ms
install_base: /var/tmp/_bazel_bdrew/install/db3d7840db9ac45fa322fb3d55e6626c
java-home: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre
java-runtime: Java(TM) SE Runtime Environment (build 1.8.0_121-b13) by Oracle Corporation
java-vm: Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) by Oracle Corporation
max-heap-size: 2512MB
message_log: /private/var/tmp/_bazel_bdrew/36485b236a991e0e17b0caecf9bb47ca/message.log
output_base: /private/var/tmp/_bazel_bdrew/36485b236a991e0e17b0caecf9bb47ca
output_path: /private/var/tmp/_bazel_bdrew/36485b236a991e0e17b0caecf9bb47ca/execroot/zippy/bazel-out
package_path: %workspace%
release: release 0.5.1-homebrew
server_pid: 26710
used-heap-size: 297MB
workspace: /Users/bdrew/devel/zippy

When I try to build this target:

py_library(
    name = "mercurylib",
    srcs = glob(["lib/**/*.py"]),
    srcs_version = "PY3",
    deps = [
        "//applications/mercury/proto:py_mercury",
        "@com_github_google_protobuf//:protobuf_python",
    ],
)

bazel tells me:

bazel build //applications/mercury:mercurylib
INFO: Reading 'startup' options from /Users/bdrew/devel/zippy/.bazelrc: --host_jvm_args=-Xmx2500m --host_jvm_args=-Xms2500m --batch
ERROR: /Users/bdrew/devel/zippy/applications/mercury/BUILD:13:1: in py_library rule //applications/mercury:mercurylib: Rule '//applications/mercury:mercurylib' need to be converted to Python 2 (not yet implemented).
ERROR: Analysis of target '//applications/mercury:mercurylib' failed; build aborted.

which is, shall we say, counterintuitive.

[brandjon]

I've been looking at Python 2 vs 3 versioning issues lately and expect to have a simple fix shortly for this kind of spurious and annoying error. Let me recap what's happening here.

The purpose of srcs_version is to annotate what major versions of Python your src files are compatible with. Today this attribute is not used to actually select what mode to build in. It'll just raise an error if you try to build it in the wrong mode.

The Python mode is controlled by --force_python (and I guess its --host... counterpart), or if that's not set, by the default_python_version attribute of a py_binary or py_test rule. Once the mode is set, it doesn't change for any further upstream targets that are depended on.

If you have a srcs_version=PY3 or PY3ONLY library, and you build a default_python_version=PY3 binary that depends on it, that'll satisfy the check. But if you build the library directly, rather than the binary, then the default_python_version attr on the binary isn't processed and the PY2 default is used instead. That's why errors occur in this situation.

The solution is to also set the python mode, if not set already, based on srcs_version if default_python_version is not applicable (as is the case for a library target). (Note that this fix does not imply that Bazel will select a python3 interpreter to use out-of-the-box. That's a separate problem.)

The original variant of the error message in this thread is obsolete. I believe it was indicating that since the Python mode was 2 but srcs_version said 3, it wanted to run 3to2 on the srcs, and 3to2 conversion was not yet implemented. Then the plan changed to not run 3to2 at all, and PY3 and PY3ONLY were made synonyms. source

[brandjon]

Update: This work was delayed by a little yak shaving to open source some unit tests for our Python rules. It's now blocked on the fact that my fix breaks some users of bazel cquery/aquery, which is experimental. We're looking at whether the underlying issue with cquery/aquery can be fixed so it's not a breakage, and if not, whether it's ok to just continue anyway. I won't be able to work on this until mid next week anyway, hopefully the blocker will be resolved by then.

[Globegitter]

@brandjon any update on this? We are kind of holding off with some of our development just until having the py2/py3 support fixed as it sounded it would not be far away. But if it is still a few weeks/months away we might have to start finding/using a different solution.

[brandjon]

When you say you're blocked on having py2/3 support fixed, which particular issues do you mean? I'm using this thread to track only the specific issue that you can't directly (i.e. on the command line) build a srcs_version = PY3 library without also passing --force_python=PY3.

I just made an index of PY2/3-related issues at #6444, most of which are targeted for this quarter.

For the bug described by this thread, here's an update. The blocker with cquery/aquery was fixed. The new blocker we're facing is that we're worried that fixing this bug will break py3 targets that have py2 data dependencies. This in turn requires fixing #6441. We may also need an incompatible change flag, but that's debatable because Python 3 support is marked "experimental". I'm anticipating resuming work on the blocker and this issue next week.

[brandjon]

Another blocker, this time more serious: We're finding that there are action conflicts between targets built in the PY2 mode, and targets built in the "I haven't decided what python version I'm using yet" mode.

That is, if you declare --force_python=PY2 or default_python_version="PY2", that's different from not declaring anything at all, even though the global default is Python 2. One example where the difference matters is if you have some transitive dependency that does a select() on "force_python", say to control what sources you're building with. Then the two modes give different results, hence the conflict when we try to write the results to the same path.

One solution is to make a separate output root to distinguish the "I haven't decided" mode from PY2 mode, the way we already have the "-py3" output root for PY3 mode.

But another solution I'm investigating is doing away with the "I haven't decided" mode altogether. The more I think about it, the more I think this model may be simpler and avoid redundant build computations.

[gpshead]

Doing away with "I haven't decided" mode sounds like the right thing.

[samuela]

In that case there would be three modes: PY2, PY3, and PY2AND3, but no "unset", correct? Then what mode would py_library default to? Like if you built a pytype_library target would it typecheck in python 2, python 3, or both? It seems that PY2AND3 is the most natural/expected choice but that's likely a breaking change for many users. And what happens when Python 4 rolls around? etc...

It seems to me that this is one big constraint solving problem. The only twist here is that some build rules change at "runtime" via select(). But perhaps a constraint solving attack could be taken here. After all it seems that using select() is a bit of a hack in the first place. For example, each target could have a set of constraints that it specifies based on its own contents, and relative to the constraints of its dependencies. Then bazel bulid would simply solve this constraint problem to identify all the configurations that are supported by each build target. Then just build all of those.

[brandjon]

PY2AND3 is a srcs_version value indicating compatibility with PY2 and PY3. It is not a value for the python mode (default_python_version / --force_python).

We have no plans to make the Python major version mode handle anything besides PY 2 vs PY 3. I believe the Python community has decided that (a hypothetical) PY 4 will be significant for its name rather than for its breaking changes, so it's unlikely we'll see the need to manage compatibility to the same extent as for 2 vs 3.

The more general problem you describe is tackled to some extent by the platforms/toolchains framework. We don't use it for Python 2 vs 3 because at the moment the framework is for platform-based constraints rather than version-based constraints, though that may change in the future.

If you want to generalize, the next issue may be minor version numbers, where you might want to say some py_library contains source that only runs on an interpreter > x.y. But this is a slippery slope: you could also be concerned about patch release versions, more expressive constraint logic (excluding one-off bad versions), constraints on the interpreter family (CPython, PyPy, etc.), and so on.

At the moment we are designing for the common use case of dealing with a dual PY2 / PY3 code base including an active migration from 2 to 3. By the time we address the other cases the platform/toolchain framework may be more of an obvious answer.

[brandjon]

--incompatible_allow_python_version_transitions is flipped in Bazel 0.25.