Cannot depend on apache-airflow with extras (e.g. [kubernetes] or [postgres,statsd,kubernetes] )

[thekyz]

🐞 bug report

Affected Rule

The issue is caused by the rule: pip_install

Is this a regression?

Tried on older versions, issue was there already.

Description

Cannot install apache-airflow

🔬 Minimal Reproduction

Code available in the repo here: https://github.com/thekyz/snippets/tree/master/rule_python_issue_with_apache-airflow

Specifically the issue is with the requirement function in here : https://github.com/thekyz/snippets/blob/b34ef1b6dcd54c26a698b0286edaed165ec4a872/rule_python_issue_with_apache-airflow/BUILD#L31

py_binary(
    name = "hello",
    srcs = [
        "hello.py",
    ],
    deps = [
        requirement("apache-airflow[postgres,statsd,kubernetes]"),
    ],
)

🔥 Exception or Error

ERROR: /home/thekyz/dev/snippets/rule_python_issue_with_apache-airflow/BUILD:25:10: no such package '@pip//pypi__apache_airflow[postgres,statsd,kubernetes]': BUILD file not found in directory 'pypi__apache_airflow[postgres,statsd,kubernetes]' of external repository @pip. Add a BUILD file to a directory to mark it as a package. and referenced by '//:hello'

🌍 Your Environment

Operating System:

Ubuntu 20.04

Output of bazel version:

Build label: 4.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jan 21 07:33:24 2021 (1611214404)
Build timestamp: 1611214404
Build timestamp as int: 1611214404

Rules_python version:

0.1.0, tried with master as well, same issue

Anything else relevant?

If I get rid of the extras in the BUILD file requirement (replace requirement("apache-airflow[postgres,statsd,kubernetes]") with `requirement("apache-airflow") I get this instead:

ERROR: /home/thekyz/.cache/bazel/_bazel_thekyz/1db85497883e553a1d95bd5ce881c422/external/pip/pypi__apache_airflow/BUILD:5:11: in py_library rule @pip//pypi__apache_airflow:pypi__apache_airflow: cycle in dependency graph:
    //:hello
.-> @pip//pypi__apache_airflow:pypi__apache_airflow
|   @pip//pypi__apache_airflow_providers_cncf_kubernetes:pypi__apache_airflow_providers_cncf_kubernetes
`-- @pip//pypi__apache_airflow:pypi__apache_airflow

[thundergolfer]

Thanks for the issue report, and thanks for doing the reproduction.

Firstly, you don't need to do requirement("apache-airflow[postgres,statsd,kubernetes]"). Once the extras are specified in the requirements file they'll be downloaded and included in requirement("apache-airflow").

The subsequent circular dependencies issue is #411. I determined that this was an error in Apache Airflow and a maintainer communicated that it would be fixed in an upcoming release.

Can you check out apache/airflow#14128 (comment) and confirm this fixes your problem?

For now I'll close this as a duplicate of #411. If this turns out to be incorrect then I'm happy to re-open and take another look.

[thekyz]

thanks, I'll check it, one small remark: the main readme makes it look like I need to reference the extras like this as this section and snippet mentions:

Consuming pip dependencies

Each extracted wheel repo contains a py_library target representing the wheel's contents. Rather than depend on this target's label directly -- which would require hardcoding the wheel repo's mangled name into your BUILD files -- you should instead use the requirement() function defined in the central repo's //:requirements.bzl file. This function maps a pip package name to a label. ("Extras" can be referenced using the pkg[extra] syntax.)

load("@my_deps//:requirements.bzl", "requirement")

py_library(
    name = "mylib",
    srcs = ["mylib.py"],
    deps = [
        ":myotherlib",
        requirement("some_pip_dep"),
        requirement("another_pip_dep[some_extra]"),
    ]
)

[thundergolfer]

Oh thanks for that heads up.