-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
0.5.3 no longer allows to enforce Python3 #3517
Comments
Printing paths before the error I get:
which is obviously wrong since I enforced Python 3:
Regression is in 0.5.4rc2, too 👎 |
There was some breaking change in 0.5.3, that unfortunately didn't make it to the release notes. I think you need to specify @meteorcloudy might know more? |
@buchgr To quote the
So I guess you mean |
So apparently, those attributes don't work anymore and you now need to specify I am in contact with relevant people... sorry :( |
It was an intentional breakage, that unfortunately did not make it into the release notes. We will provide a blog post and updated documentation soon. |
Can you perhaps write a bit more here. |
My understanding is that starting with 0.5.3, bazel no longer knows or cares about the python version. You just point it to a python runtime and it will use it. The |
Sadly that is wrong.
That may be correct. Please either remove them or add tests to validate every permutation. Still there is Also it seems like Can you please make the dev in charge of Python write up how it is "supposed" to work now and I will file bugs then. |
Is it then no longer possible to have both Python2 and Python3 in one repo? |
Any movement on this? |
I think I just ran into that when attempting to upgrade from 0.5.0 to 0.6.0 yesterday. We've got mixed python2 and python3 binaries in our build. I'd love to upgrade to python3 entirely, but not all of the libraries we use support python3 yet. |
@lberki can tell you more about plans for python maybe? |
At the very minimum, I need to be able to have a single codebase with both Python 2 and Python 3 code, and then manually specify in the individual test/binary which interpreter to use. Is this possible somehow? A temporary workaround would be fine. A global flag is unsuitable, since I need to be able to do e.g. (This is in particular necessitated by Google-developed Python 2-only libraries like Apache Beam.) |
@lberki can you take this please? |
Sooo soonish, we will be a year later and still no explanation? |
friendly ping @lberki :) |
I just wanted to offer a temporary workaround that we're using to allow multiple python runtimes in a single workspace. You can set your py_runtime to a decorator script that inspects the shebang on the first line of the script to invoke a particular interpreter. You could make this a simple shell script which someone had suggested in another thread I can't seem to find. If you're using distroless images (py_image or py3_image) using a shell script might be an issue so I wrote a statically linked go application. This is pretty heavy handed but not so bad if you're already using go in your workspace. Hopefully either one of these options is viable until someone gets this sorted out. .bazelrc
BUILD load(
"@io_bazel_rules_go//go:def.bzl",
"go_binary",
)
go_binary(
name = "python_decorator",
srcs = ["python.go"],
gc_linkopts = [
"-linkmode",
"external",
"-extldflags",
"-static",
],
)
py_runtime(
name = "python-2-or-3",
files = [],
interpreter = ":python_decorator",
) python.go package main
import (
"bufio"
"log"
"os"
"strings"
"syscall"
)
func python2Binary() string {
return "/usr/bin/python"
}
func python3Binary() string {
return "/usr/bin/python3"
}
func main() {
if len(os.Args) < 2 {
log.Fatal("No python script provided!")
}
file, err := os.Open(os.Args[1])
if err != nil {
log.Fatal(err)
}
defer file.Close()
scanner := bufio.NewScanner(file)
scanner.Scan()
firstLine := scanner.Text()
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
switch {
case strings.Contains(firstLine, "python3"):
if err := syscall.Exec(python3Binary(), append([]string{python3Binary()}, os.Args[1:]...), os.Environ()); err != nil {
log.Fatal(err)
}
default:
if err := syscall.Exec(python2Binary(), append([]string{python2Binary()}, os.Args[1:]...), os.Environ()); err != nil {
log.Fatal(err)
}
}
} |
Thanks @AdamDorwart ! |
Another way of doing it is using a It still requires a custom arg to bazel; imho bazel should have it configured like this by default. Maybe an easy fix is to just add the config_setting() and select() I mention below directly to whatever Bazel's default py_runtime definition is. Also, some caveats:
Also, just for posterity and to reconfirm what someone said up thread: Reading through the Bazel source, it looks like only --python_path and py_runtime are used to figure out the binary to use to run the python target (i started with how it populates the template script and went backwards from there). I'm guessing this is because Bazel expects people to use the select() way to select different values depending on the configuration. |
@AdamDorwart I believe this is the bash script you were referencing: if head -n 1 "$1" | grep -q python3; then
exec "$BASE_PATH"/usr/bin/python3 "$@"
else
exec "$BASE_PATH"/usr/bin/python2 "$@"
fi from https://groups.google.com/forum/#!topic/bazel-discuss/nVQ48R94S_8 |
Note that the shebang-parsing way will choose the version based on the actual source file contents, while the select() way (and the eventual fix to Bazel's Python rules) will choose based on the version selected in the build (default_python_version attr / --force_python). @rickeylev's right about the transition being one way, but I can imagine changing it so that it's only one way through deps, and gets reset when it crosses a data attribute. The host mode issue is also a problem because it means you can't use Python 3 tools in your genrules (or if you invert it with --host_force_python=PY3, then you can't use Python 2 tools). That one's harder to fix, so if you really need multiple python versions in your host config you may want to stick with indirecting through a script. |
I'm closing this issue and opening several more directed ones to cover the points raised in this thread.
For a more detailed index of Python 2/3 issues, see tracking bug #6444, most of which is targeted for this quarter. |
Note that |
Description of the problem / feature request / question:
I have a python script, that worked fine in 0.5.2.
Switching to 0.5.3 running
gives me:
Environment info
bazel info release
): 0.5.3The text was updated successfully, but these errors were encountered: