Fetching dependencies assumes python2

[hwright]

Specifically, see this line:

containerregistry/def.bzl

Line 20 in 60a61ec

strip_prefix = "httplib2-0.10.3/python2/httplib2/",

The httplib2 package ships two separate trees for python 2/3 compat, and we are exclusively using the python2 tree. This means that if a user is trying to run this code under python 3, it will fail, since that tree is not compatible with python3.

I'm not sure the right answer here. In an ideal world, Bazel would expose the python version to us, and we could include the right tree based on that. Perhaps we could fix bazelbuild/rules_python#33 and then use pip to import these dependencies, rather than fetching the archives directly.

Alternatively, we could move from httplib2 to urllib (or the python 2/3 compatible version in six.moves.urllib), but it turns out that oauth2client also uses httplib2, so we'd still need it for transitive dependency resolution.

Until this is fixed, it is unlikely that python 3 users can use these rules.

[hwright]

Fun fact: According to https://github.com/google/oauth2client, "oauth2client is now deprecated." And one of the reasons for deprecation is the dependency on httplib2!

The recommended replacement is google-auth. I'd dig in and do the replacement, but there seems to be enough Google-specific sauce going on that somebody on the core team might want to take a look.

[mytran]

Do you know when python3 will be officially supported? We're stuck on a really old version of Bazel since the newer versions of Bazel run these rules with whatever you have your py_runtime set to.

[kalbasit]

@mytran try setting the BAZEL_PYTHON environment variable to the absolute path of python2. This worked for me.

[kalbasit]

@mytran actually that helped only for building the image, but it still failed while trying to push the image. Here's another workaround:

mkdir ~/path-for-python2-default
cd ~/path-for-python2-default
ln -s /usr/bin/python2 python

export PATH=~/path-for-python2-default:$PATH
bazel [...]

By putting that hacky path first in the PATH variable, bazel and all it's dependency will see the symling python which is python2.

[mytran]

I got stuck on building the container_image.

Hit a snag with the debian packages in container_image __main__.DebError: external/libc6_deb/file/libc6_2.23-0ubuntu10_amd64.deb contains invalid Metadata! Exeception 'filter' object is not subscriptable. I removed the debs section to get around it and got another import error:

__main__/bazel-out/host/bin/external/io_bazel_rules_docker/container/join_layers.runfiles/containerregistry/client/docker_name_.py", line 21, in <module>
    import urlparse
ModuleNotFoundError: No module named 'urlparse'

[jmhodges]

I've run into that urlparse error, as well, when trying to build containers with bazel's rules_docker library. (See bazelbuild/rules_docker#293)

[Globegitter]

#85 brought python 3 support and there is a PR for rules_docker (bazelbuild/rules_docker#449), so can this be closed now?

[whilp]

@Globegitter I do not believe this should be closed yet, as the puller.par is still produced with the python2-only httplib2 mentioned in the body of the issue:

containerregistry/def.bzl

Line 20 in 2f1cfd3

strip_prefix = "httplib2-0.10.3/python2/httplib2/",

Despite #85 and bazelbuild/rules_docker#449, users that attempt to run bazel with Python 3 will still be unable to pull images. For example, I get the following:

  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/cache/bazel/_bazel_root/c0b43bcf2fdefc559ca549fb22bb7e37/external/puller/file/downloaded/__main__.py", line 30, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/cache/bazel/_bazel_root/c0b43bcf2fdefc559ca549fb22bb7e37/external/puller/file/downloaded/containerregistry/client/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/cache/bazel/_bazel_root/c0b43bcf2fdefc559ca549fb22bb7e37/external/puller/file/downloaded/containerregistry/client/docker_creds_.py", line 31, in <module>
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 894, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1131, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1112, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 441, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 544, in spec_from_file_location
  File "/cache/bazel/_bazel_root/c0b43bcf2fdefc559ca549fb22bb7e37/external/puller/file/downloaded/httplib2/__init__.py", line 942
    print "connect: (%s, %s) ************" % (self.host, self.port)
                                         ^
SyntaxError: invalid syntax (/usr/bin/python /cache/bazel/_bazel_root/c0b43bcf2fdefc559ca549fb22bb7e37/external/puller/file/downloaded --directory /cache/bazel/_bazel_root/c0b43bcf2fdefc559ca549fb22bb7e37/external/ubuntu/image --name index.docker.io/library/ubuntu@sha256:30e04ddada6eb09c12330c7df72cad1573916c7100168c34076808169ff6d805) and referenced by '//image:image'

[dekkagaijin]

+cc @deft-code
Someone could 'shim' over the py 2/3 versions of httplib2, containerregistry itself works with either version.

[whilp]

One step forward seems close here:
bazelbuild/bazel#6532