Support transitive dependencies

[hwright]

https://docs.bazel.build/versions/master/external.html#transitive-dependencies explicitly says that WORKSPACE files must include all transitive dependencies for a project. That make sense, given the very really possibility of conflicting version requirements for dependent projects.

However, one of the promises of pip is that transitive dependencies are resolved and installed automatically. With the Bazel python rules, using an external package which internally uses pip_import and friends means that we loose the transitive dependency resolution of pip for that package. The conflicting version problem hasn't gone away, since it is still possible it could occur with the pip_import-managed packages, we've just made life harder on ourselves.

It would be nice to have a way of pulling down the transitive dependencies for external Bazel projects using the pip_import infrastructure. This could be in the form of generate_workspace from Java-land, or it could be a way to pull down those dependencies and create a master requirements file, similar to pip-compile. In any case, manually managing dependencies for external Bazel projects is going to get unwieldy very quickly.

[duggelz]

See bazelbuild/bazel#1943 from 2016. There was a Blaze Summit session about recursive workspaces yesterday. Perhaps @mattmoor can provide a status update.

[duggelz]

Some recent discussion in this doc: https://docs.google.com/document/d/1HfRGRW4MwnVUG24rw3HJIBkdEYfXlESnlKOboO97A3A/edit#heading=h.weiamv45ubqe

[hwright]

Thanks.

I should also note that with pip_import, the transitive dependency problem already exists today, because those dependencies pull down from pypi may have conflicts among themselves. With pip_import, we loose the ability to resolve transitive dependencies for external Bazel repos, but don't really eliminate the problem from other areas. It's currently the worst of both worlds.

[mattmoor]

Took me a while to parse what you are getting at, but now that I think I understand.... and I'll raise you something more ambitious :)

What would it take for use to be able to be able to support entries in requirements.txt like:

git://github.com/foo-bar/bazel-project.git#egg=asdf

(see also)

This is essentially what K8s is doing right now: maintaining compat between go build and bazel build. @ixdy FYI

I don't have a good answer for this (I don't think anybody does), but I could see us exposing a runnable target that could translate a pip_import into a setup.py that folks could check into their release branch? In fact, we could also generate a py_test target that checks whether a checked in setup.py is out of date (similar to what I was going for in this PR) :)

There is significant duality here to a problem I've been putting together a document about in terms of Bazel migration, and it would make sense for both the Bazel ingress and egress to be managed by common tooling per tool/language (e.g. pip, Go, mvn, ...).

For the more basic (as if it weren't hard enough) "transitive deps in Bazel", @damienmg is the guy (IIRC).

[cyanfish]

For anyone else trying to google why transitive pip dependencies aren't working within a single project, the relevant issue is #125 (which has instructions for using an unofficial build with a fix).

[alexeagle]

rules_python 0.1.0 has been released which upstreams the rules_python_external repo. Please switch from pip_import to pip_install which doesn't have this issue.