Problem handling namespace packages with circular dependencies in Airflow 2 #411
Description
I'm not sure if this is technically a bug with rules_python, but I've run into some trouble installing airflow 2. I thought I would open here for advice and to surface the issue for others.
Example repo to repro the failure: https://github.com/gvacaliuc/rules_python_airflow2
The apache-airflow pypi package now depends on a few other small namespace packages which in turn depend on apache-airflow. pip apparently resolves this cyclic dependency, but rules_python produces an invalid library structure:
gvacaliuc@chiron:~/.../gvacaliuc/rules_python_airflow2$ cat /home/gvacaliuc/.cache/bazel/_bazel_gvacaliuc/85385b6bcbbb23d240857dfefd8684dd/external/py_deps/pypi__apache_airflow/BUILD
package(default_visibility = ["//visibility:public"])
load("@rules_python//python:defs.bzl", "py_library")
py_library(
name = "pypi__apache_airflow",
srcs = glob(["**/*.py"], allow_empty = True),
data = glob(["**/*"], exclude=["**/*.py", "**/* *", "BUILD", "WORKSPACE"]),
# This makes this directory a top-level in the python import
# search path for anything that depends on this.
imports = ["."],
deps = ["//pypi__alembic","//pypi__apache_airflow_providers_ftp","//pypi__apache_airflow_providers_http","//pypi__apache_airflow_providers_imap","//pypi__apache_airflow_providers_sqlite","//pypi__argcomplete","//pypi__attrs","//pypi__cached_property","//pypi__cattrs","//pypi__colorlog","//pypi__connexion","//pypi__croniter","//pypi__cryptography","//pypi__dill","//pypi__flask","//pypi__flask_appbuilder","//pypi__flask_caching","//pypi__flask_login","//pypi__flask_swagger","//pypi__flask_wtf","//pypi__funcsigs","//pypi__graphviz","//pypi__gunicorn","//pypi__importlib_metadata","//pypi__importlib_resources","//pypi__iso8601","//pypi__itsdangerous","//pypi__jinja2","//pypi__json_merge_patch","//pypi__jsonschema","//pypi__lazy_object_proxy","//pypi__lockfile","//pypi__markdown","//pypi__markupsafe","//pypi__marshmallow_oneofschema","//pypi__pandas","//pypi__pendulum","//pypi__psutil","//pypi__pygments","//pypi__python_daemon","//pypi__python_dateutil","//pypi__python_nvd3","//pypi__python_slugify","//pypi__requests","//pypi__rich","//pypi__setproctitle","//pypi__sqlalchemy","//pypi__sqlalchemy_jsonfield","//pypi__tabulate","//pypi__tenacity","//pypi__termcolor","//pypi__thrift","//pypi__tzlocal","//pypi__unicodecsv","//pypi__urllib3","//pypi__werkzeug"],
)
gvacaliuc@chiron:~/.../gvacaliuc/rules_python_airflow2$ cat /home/gvacaliuc/.cache/bazel/_bazel_gvacaliuc/85385b6bcbbb23d240857dfefd8684dd/external/py_deps/pypi__apache_airflow_providers_sqlite/BUILD
package(default_visibility = ["//visibility:public"])
load("@rules_python//python:defs.bzl", "py_library")
py_library(
name = "pypi__apache_airflow_providers_sqlite",
srcs = glob(["**/*.py"], allow_empty = True),
data = glob(["**/*"], exclude=["**/*.py", "**/* *", "BUILD", "WORKSPACE"]),
# This makes this directory a top-level in the python import
# search path for anything that depends on this.
imports = ["."],
deps = ["//pypi__apache_airflow"],
)
Resulting in the build failure:
gvacaliuc@chiron:~/.../gvacaliuc/rules_python_airflow2$ bazelisk build :main
ERROR: /home/gvacaliuc/.cache/bazel/_bazel_gvacaliuc/85385b6bcbbb23d240857dfefd8684dd/external/py_deps/pypi__apache_airflow/BUILD:5:1: in py_library rule @py_deps//pypi__apache_airflow:pypi__apache_airflow: cycle in dependency graph:
//:main
.-> @py_deps//pypi__apache_airflow:pypi__apache_airflow
| @py_deps//pypi__apache_airflow_providers_sqlite:pypi__apache_airflow_providers_sqlite
`-- @py_deps//pypi__apache_airflow:pypi__apache_airflow
This cycle occurred because of a configuration option
ERROR: Analysis of target '//:main' failed; build aborted
INFO: Elapsed time: 0.159s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)
In this case, it seems like we'd want to merge the contents of the airflow provider wheels with the main airflow wheel before generating the build files?
Metadata
Metadata
Assignees
Labels
No labels